2023-04-27 01:53:32,146 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7 2023-04-27 01:53:32,159 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-27 01:53:32,176 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-27 01:53:32,176 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/cluster_f8bfb6d3-6781-ae77-a03b-b65e0afd0224, deleteOnExit=true 2023-04-27 01:53:32,177 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-27 01:53:32,177 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/test.cache.data in system properties and HBase conf 2023-04-27 01:53:32,178 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/hadoop.tmp.dir in system properties and HBase conf 2023-04-27 01:53:32,178 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/hadoop.log.dir in system properties and HBase conf 2023-04-27 01:53:32,179 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-27 01:53:32,180 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-27 01:53:32,180 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-27 01:53:32,296 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-27 01:53:32,725 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-27 01:53:32,731 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-27 01:53:32,732 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-27 01:53:32,732 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-27 01:53:32,733 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-27 01:53:32,733 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-27 01:53:32,734 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-27 01:53:32,734 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-27 01:53:32,734 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-27 01:53:32,735 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-27 01:53:32,735 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/nfs.dump.dir in system properties and HBase conf 2023-04-27 01:53:32,736 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/java.io.tmpdir in system properties and HBase conf 2023-04-27 01:53:32,736 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-27 01:53:32,737 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-27 01:53:32,737 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-27 01:53:33,343 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-27 01:53:33,347 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-27 01:53:33,674 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-27 01:53:33,863 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-27 01:53:33,883 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-27 01:53:33,929 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-27 01:53:33,960 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/java.io.tmpdir/Jetty_localhost_46651_hdfs____.p0ykzq/webapp 2023-04-27 01:53:34,109 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46651 2023-04-27 01:53:34,126 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-27 01:53:34,126 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-27 01:53:34,726 WARN [Listener at localhost/44221] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-27 01:53:34,793 WARN [Listener at localhost/44221] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-27 01:53:34,812 WARN [Listener at localhost/44221] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-27 01:53:34,819 INFO [Listener at localhost/44221] log.Slf4jLog(67): jetty-6.1.26 2023-04-27 01:53:34,824 INFO [Listener at localhost/44221] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/java.io.tmpdir/Jetty_localhost_33099_datanode____.bfhauh/webapp 2023-04-27 01:53:34,932 INFO [Listener at localhost/44221] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33099 2023-04-27 01:53:35,288 WARN [Listener at localhost/42495] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-27 01:53:35,832 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcccc1b9481f2c092: Processing first storage report for DS-7b0e825c-f84c-4211-847f-62201de12276 from datanode 82e9d633-1016-4634-b79e-7e0d7bfee16d 2023-04-27 01:53:35,833 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcccc1b9481f2c092: from storage DS-7b0e825c-f84c-4211-847f-62201de12276 node DatanodeRegistration(127.0.0.1:36053, datanodeUuid=82e9d633-1016-4634-b79e-7e0d7bfee16d, infoPort=33561, infoSecurePort=0, ipcPort=42495, storageInfo=lv=-57;cid=testClusterID;nsid=975706845;c=1682560413421), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-27 01:53:35,833 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcccc1b9481f2c092: Processing first storage report for DS-77bf42a1-b215-4e34-b9cd-240e914e86d3 from datanode 82e9d633-1016-4634-b79e-7e0d7bfee16d 2023-04-27 01:53:35,834 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcccc1b9481f2c092: from storage DS-77bf42a1-b215-4e34-b9cd-240e914e86d3 node DatanodeRegistration(127.0.0.1:36053, datanodeUuid=82e9d633-1016-4634-b79e-7e0d7bfee16d, infoPort=33561, infoSecurePort=0, ipcPort=42495, storageInfo=lv=-57;cid=testClusterID;nsid=975706845;c=1682560413421), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-27 01:53:35,875 DEBUG [Listener at localhost/42495] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7 2023-04-27 01:53:35,935 INFO [Listener at localhost/42495] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/cluster_f8bfb6d3-6781-ae77-a03b-b65e0afd0224/zookeeper_0, clientPort=52731, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/cluster_f8bfb6d3-6781-ae77-a03b-b65e0afd0224/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/cluster_f8bfb6d3-6781-ae77-a03b-b65e0afd0224/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-27 01:53:35,947 INFO [Listener at localhost/42495] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52731 2023-04-27 01:53:35,956 INFO [Listener at localhost/42495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 01:53:35,958 INFO [Listener at localhost/42495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 01:53:36,634 INFO [Listener at localhost/42495] util.FSUtils(471): Created version file at hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b with version=8 2023-04-27 01:53:36,634 INFO [Listener at localhost/42495] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/hbase-staging 2023-04-27 01:53:36,964 INFO [Listener at localhost/42495] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-27 01:53:37,427 INFO [Listener at localhost/42495] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-27 01:53:37,461 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 01:53:37,462 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-27 01:53:37,462 INFO [Listener at localhost/42495] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-27 01:53:37,462 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 01:53:37,462 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-27 01:53:37,620 INFO [Listener at localhost/42495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-27 01:53:37,692 DEBUG [Listener at localhost/42495] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-27 01:53:37,786 INFO [Listener at localhost/42495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34643 2023-04-27 01:53:37,796 INFO [Listener at localhost/42495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 01:53:37,798 INFO [Listener at localhost/42495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 01:53:37,820 INFO [Listener at localhost/42495] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34643 connecting to ZooKeeper ensemble=127.0.0.1:52731 2023-04-27 01:53:37,861 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:346430x0, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-27 01:53:37,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34643-0x101c35262760000 connected 2023-04-27 01:53:37,892 DEBUG [Listener at localhost/42495] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-27 01:53:37,893 DEBUG [Listener at localhost/42495] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 01:53:37,896 DEBUG [Listener at localhost/42495] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-27 01:53:37,904 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34643 2023-04-27 01:53:37,904 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34643 2023-04-27 01:53:37,905 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34643 2023-04-27 01:53:37,905 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34643 2023-04-27 01:53:37,905 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34643 2023-04-27 01:53:37,911 INFO [Listener at localhost/42495] master.HMaster(444): hbase.rootdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b, hbase.cluster.distributed=false 2023-04-27 01:53:37,986 INFO [Listener at localhost/42495] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-27 01:53:37,986 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 01:53:37,987 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-27 01:53:37,987 INFO [Listener at localhost/42495] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-27 01:53:37,987 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 01:53:37,987 INFO [Listener at localhost/42495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-27 01:53:37,992 INFO [Listener at localhost/42495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-27 01:53:37,995 INFO [Listener at localhost/42495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38085 2023-04-27 01:53:37,997 INFO [Listener at localhost/42495] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-27 01:53:38,003 DEBUG [Listener at localhost/42495] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-27 01:53:38,004 INFO [Listener at localhost/42495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 01:53:38,006 INFO [Listener at localhost/42495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 01:53:38,008 INFO [Listener at localhost/42495] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38085 connecting to ZooKeeper ensemble=127.0.0.1:52731 2023-04-27 01:53:38,011 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:380850x0, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-27 01:53:38,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38085-0x101c35262760001 connected 2023-04-27 01:53:38,013 DEBUG [Listener at localhost/42495] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-27 01:53:38,014 DEBUG [Listener at localhost/42495] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 01:53:38,015 DEBUG [Listener at localhost/42495] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-27 01:53:38,015 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38085 2023-04-27 01:53:38,015 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38085 2023-04-27 01:53:38,016 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38085 2023-04-27 01:53:38,017 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38085 2023-04-27 01:53:38,017 DEBUG [Listener at localhost/42495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38085 2023-04-27 01:53:38,019 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:38,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-27 01:53:38,032 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:38,050 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-27 01:53:38,050 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-27 01:53:38,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:38,052 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-27 01:53:38,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34643,1682560416787 from backup master directory 2023-04-27 01:53:38,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-27 01:53:38,057 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:38,057 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-27 01:53:38,058 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-27 01:53:38,058 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:38,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-27 01:53:38,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-27 01:53:38,554 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/hbase.id with ID: 377279bf-da22-4fcf-aa2f-92e691319a52 2023-04-27 01:53:38,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 01:53:38,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:38,652 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7731e96a to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:38,688 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ae4435, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:38,714 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:53:38,716 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-27 01:53:38,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-27 01:53:38,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-27 01:53:38,739 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:53:38,743 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:53:38,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-27 01:53:38,780 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store-tmp 2023-04-27 01:53:38,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:38,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-27 01:53:38,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:53:38,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:53:38,814 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-27 01:53:38,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:53:38,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:53:38,815 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-27 01:53:38,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/WALs/jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:38,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34643%2C1682560416787, suffix=, logDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/WALs/jenkins-hbase4.apache.org,34643,1682560416787, archiveDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/oldWALs, maxLogs=10 2023-04-27 01:53:38,896 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36053,DS-7b0e825c-f84c-4211-847f-62201de12276,DISK] 2023-04-27 01:53:38,904 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:53:38,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/WALs/jenkins-hbase4.apache.org,34643,1682560416787/jenkins-hbase4.apache.org%2C34643%2C1682560416787.1682560418853 2023-04-27 01:53:38,974 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36053,DS-7b0e825c-f84c-4211-847f-62201de12276,DISK]] 2023-04-27 01:53:38,975 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:53:38,975 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:38,978 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-27 01:53:38,980 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-27 01:53:39,046 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-27 01:53:39,054 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-27 01:53:39,079 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-27 01:53:39,090 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:39,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-27 01:53:39,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-27 01:53:39,115 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-27 01:53:39,119 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:53:39,121 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60573310, jitterRate=-0.09738734364509583}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-27 01:53:39,121 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-27 01:53:39,122 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-27 01:53:39,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-27 01:53:39,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-27 01:53:39,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-27 01:53:39,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-27 01:53:39,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-27 01:53:39,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-27 01:53:39,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-27 01:53:39,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-27 01:53:39,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-27 01:53:39,242 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-27 01:53:39,244 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-27 01:53:39,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-27 01:53:39,255 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-27 01:53:39,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:39,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-27 01:53:39,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-27 01:53:39,276 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-27 01:53:39,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-27 01:53:39,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-27 01:53:39,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:39,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34643,1682560416787, sessionid=0x101c35262760000, setting cluster-up flag (Was=false) 2023-04-27 01:53:39,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:39,309 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-27 01:53:39,311 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:39,316 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:39,324 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-27 01:53:39,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:39,329 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.hbase-snapshot/.tmp 2023-04-27 01:53:39,423 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(952): ClusterId : 377279bf-da22-4fcf-aa2f-92e691319a52 2023-04-27 01:53:39,429 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-27 01:53:39,439 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-27 01:53:39,439 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-27 01:53:39,443 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-27 01:53:39,445 DEBUG [RS:0;jenkins-hbase4:38085] zookeeper.ReadOnlyZKClient(139): Connect 0x199fd150 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:39,458 DEBUG [RS:0;jenkins-hbase4:38085] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17582c13, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:39,459 DEBUG [RS:0;jenkins-hbase4:38085] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1367ba20, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-27 01:53:39,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-27 01:53:39,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 01:53:39,481 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 01:53:39,481 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 01:53:39,482 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 01:53:39,482 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-27 01:53:39,482 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-27 01:53:39,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,485 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:38085 2023-04-27 01:53:39,489 INFO [RS:0;jenkins-hbase4:38085] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-27 01:53:39,490 INFO [RS:0;jenkins-hbase4:38085] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-27 01:53:39,490 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1023): About to register with Master. 2023-04-27 01:53:39,490 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682560449490 2023-04-27 01:53:39,493 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-27 01:53:39,494 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,34643,1682560416787 with isa=jenkins-hbase4.apache.org/172.31.14.131:38085, startcode=1682560417985 2023-04-27 01:53:39,498 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-27 01:53:39,498 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-27 01:53:39,504 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-27 01:53:39,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-27 01:53:39,518 DEBUG [RS:0;jenkins-hbase4:38085] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-27 01:53:39,521 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-27 01:53:39,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-27 01:53:39,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-27 01:53:39,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-27 01:53:39,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,526 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-27 01:53:39,527 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-27 01:53:39,528 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-27 01:53:39,531 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-27 01:53:39,531 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-27 01:53:39,533 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682560419533,5,FailOnTimeoutGroup] 2023-04-27 01:53:39,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682560419536,5,FailOnTimeoutGroup] 2023-04-27 01:53:39,536 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,536 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-27 01:53:39,538 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,538 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,577 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53359, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-27 01:53:39,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:39,608 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b 2023-04-27 01:53:39,609 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44221 2023-04-27 01:53:39,609 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-27 01:53:39,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-27 01:53:39,617 DEBUG [RS:0;jenkins-hbase4:38085] zookeeper.ZKUtil(162): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:39,617 WARN [RS:0;jenkins-hbase4:38085] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-27 01:53:39,618 INFO [RS:0;jenkins-hbase4:38085] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-27 01:53:39,619 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/WALs/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:39,619 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38085,1682560417985] 2023-04-27 01:53:39,637 DEBUG [RS:0;jenkins-hbase4:38085] zookeeper.ZKUtil(162): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:39,648 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-27 01:53:39,658 INFO [RS:0;jenkins-hbase4:38085] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-27 01:53:39,679 INFO [RS:0;jenkins-hbase4:38085] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-27 01:53:39,682 INFO [RS:0;jenkins-hbase4:38085] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-27 01:53:39,682 INFO [RS:0;jenkins-hbase4:38085] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,684 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-27 01:53:39,691 INFO [RS:0;jenkins-hbase4:38085] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,691 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,692 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,692 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,692 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,692 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,693 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-27 01:53:39,693 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,693 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,693 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,694 DEBUG [RS:0;jenkins-hbase4:38085] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 01:53:39,696 INFO [RS:0;jenkins-hbase4:38085] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,696 INFO [RS:0;jenkins-hbase4:38085] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,696 INFO [RS:0;jenkins-hbase4:38085] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,716 INFO [RS:0;jenkins-hbase4:38085] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-27 01:53:39,719 INFO [RS:0;jenkins-hbase4:38085] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38085,1682560417985-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:39,744 INFO [RS:0;jenkins-hbase4:38085] regionserver.Replication(203): jenkins-hbase4.apache.org,38085,1682560417985 started 2023-04-27 01:53:39,744 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38085,1682560417985, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38085, sessionid=0x101c35262760001 2023-04-27 01:53:39,744 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-27 01:53:39,745 DEBUG [RS:0;jenkins-hbase4:38085] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:39,745 DEBUG [RS:0;jenkins-hbase4:38085] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38085,1682560417985' 2023-04-27 01:53:39,745 DEBUG [RS:0;jenkins-hbase4:38085] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:39,746 DEBUG [RS:0;jenkins-hbase4:38085] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:39,746 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-27 01:53:39,746 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-27 01:53:39,746 DEBUG [RS:0;jenkins-hbase4:38085] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:39,746 DEBUG [RS:0;jenkins-hbase4:38085] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38085,1682560417985' 2023-04-27 01:53:39,746 DEBUG [RS:0;jenkins-hbase4:38085] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-27 01:53:39,747 DEBUG [RS:0;jenkins-hbase4:38085] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-27 01:53:39,748 DEBUG [RS:0;jenkins-hbase4:38085] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-27 01:53:39,748 INFO [RS:0;jenkins-hbase4:38085] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-27 01:53:39,748 INFO [RS:0;jenkins-hbase4:38085] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-27 01:53:39,860 INFO [RS:0;jenkins-hbase4:38085] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38085%2C1682560417985, suffix=, logDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/WALs/jenkins-hbase4.apache.org,38085,1682560417985, archiveDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/oldWALs, maxLogs=32 2023-04-27 01:53:39,885 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36053,DS-7b0e825c-f84c-4211-847f-62201de12276,DISK] 2023-04-27 01:53:39,890 INFO [RS:0;jenkins-hbase4:38085] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/WALs/jenkins-hbase4.apache.org,38085,1682560417985/jenkins-hbase4.apache.org%2C38085%2C1682560417985.1682560419863 2023-04-27 01:53:39,890 DEBUG [RS:0;jenkins-hbase4:38085] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36053,DS-7b0e825c-f84c-4211-847f-62201de12276,DISK]] 2023-04-27 01:53:39,950 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-27 01:53:39,951 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-27 01:53:39,951 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b 2023-04-27 01:53:39,964 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:39,966 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-27 01:53:39,969 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/info 2023-04-27 01:53:39,970 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-27 01:53:39,971 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:39,971 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-27 01:53:39,973 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/rep_barrier 2023-04-27 01:53:39,974 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-27 01:53:39,975 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:39,975 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-27 01:53:39,977 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/table 2023-04-27 01:53:39,978 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-27 01:53:39,979 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:39,980 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740 2023-04-27 01:53:39,981 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740 2023-04-27 01:53:39,984 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:53:39,986 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-27 01:53:39,989 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:53:39,990 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67459061, jitterRate=0.005218341946601868}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:53:39,990 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-27 01:53:39,990 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-27 01:53:39,990 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-27 01:53:39,990 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-27 01:53:39,990 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-27 01:53:39,991 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-27 01:53:39,991 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-27 01:53:39,992 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-27 01:53:39,997 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-27 01:53:39,997 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-27 01:53:40,005 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-27 01:53:40,017 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-27 01:53:40,020 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-27 01:53:40,172 DEBUG [jenkins-hbase4:34643] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-27 01:53:40,176 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38085,1682560417985, state=OPENING 2023-04-27 01:53:40,183 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-27 01:53:40,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:40,186 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-27 01:53:40,188 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:53:40,373 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:40,375 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-27 01:53:40,379 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-27 01:53:40,390 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-27 01:53:40,390 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-27 01:53:40,394 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38085%2C1682560417985.meta, suffix=.meta, logDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/WALs/jenkins-hbase4.apache.org,38085,1682560417985, archiveDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/oldWALs, maxLogs=32 2023-04-27 01:53:40,413 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36053,DS-7b0e825c-f84c-4211-847f-62201de12276,DISK] 2023-04-27 01:53:40,418 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/WALs/jenkins-hbase4.apache.org,38085,1682560417985/jenkins-hbase4.apache.org%2C38085%2C1682560417985.meta.1682560420395.meta 2023-04-27 01:53:40,418 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36053,DS-7b0e825c-f84c-4211-847f-62201de12276,DISK]] 2023-04-27 01:53:40,419 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:53:40,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-27 01:53:40,443 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-27 01:53:40,448 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-27 01:53:40,453 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-27 01:53:40,453 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:40,453 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-27 01:53:40,454 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-27 01:53:40,456 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-27 01:53:40,458 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/info 2023-04-27 01:53:40,458 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/info 2023-04-27 01:53:40,459 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-27 01:53:40,460 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:40,461 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-27 01:53:40,462 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/rep_barrier 2023-04-27 01:53:40,462 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/rep_barrier 2023-04-27 01:53:40,463 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-27 01:53:40,464 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:40,464 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-27 01:53:40,466 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/table 2023-04-27 01:53:40,466 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/table 2023-04-27 01:53:40,466 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-27 01:53:40,467 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:40,469 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740 2023-04-27 01:53:40,472 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740 2023-04-27 01:53:40,476 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:53:40,479 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-27 01:53:40,481 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61836055, jitterRate=-0.07857097685337067}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:53:40,481 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-27 01:53:40,492 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682560420365 2023-04-27 01:53:40,510 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-27 01:53:40,511 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-27 01:53:40,512 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38085,1682560417985, state=OPEN 2023-04-27 01:53:40,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-27 01:53:40,517 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-27 01:53:40,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-27 01:53:40,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38085,1682560417985 in 329 msec 2023-04-27 01:53:40,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-27 01:53:40,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 518 msec 2023-04-27 01:53:40,532 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1530 sec 2023-04-27 01:53:40,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682560420532, completionTime=-1 2023-04-27 01:53:40,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-27 01:53:40,534 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-27 01:53:40,594 DEBUG [hconnection-0x4ebc138c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:40,597 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:40,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-27 01:53:40,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682560480614 2023-04-27 01:53:40,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682560540614 2023-04-27 01:53:40,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 80 msec 2023-04-27 01:53:40,635 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34643,1682560416787-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:40,636 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34643,1682560416787-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:40,636 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34643,1682560416787-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:40,637 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34643, period=300000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:40,638 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-27 01:53:40,644 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-27 01:53:40,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-27 01:53:40,654 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-27 01:53:40,663 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-27 01:53:40,665 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 01:53:40,667 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 01:53:40,689 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:40,691 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a empty. 2023-04-27 01:53:40,692 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:40,692 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-27 01:53:41,115 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-27 01:53:41,117 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8b6eb92cd0f6859befa2a3fe078eb91a, NAME => 'hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp 2023-04-27 01:53:41,530 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:41,530 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 8b6eb92cd0f6859befa2a3fe078eb91a, disabling compactions & flushes 2023-04-27 01:53:41,530 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:53:41,530 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:53:41,530 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. after waiting 0 ms 2023-04-27 01:53:41,530 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:53:41,530 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:53:41,530 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 8b6eb92cd0f6859befa2a3fe078eb91a: 2023-04-27 01:53:41,535 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 01:53:41,549 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682560421538"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560421538"}]},"ts":"1682560421538"} 2023-04-27 01:53:41,574 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 01:53:41,575 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 01:53:41,580 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560421576"}]},"ts":"1682560421576"} 2023-04-27 01:53:41,583 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-27 01:53:41,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8b6eb92cd0f6859befa2a3fe078eb91a, ASSIGN}] 2023-04-27 01:53:41,595 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=8b6eb92cd0f6859befa2a3fe078eb91a, ASSIGN 2023-04-27 01:53:41,597 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=8b6eb92cd0f6859befa2a3fe078eb91a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=false 2023-04-27 01:53:41,748 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8b6eb92cd0f6859befa2a3fe078eb91a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:41,748 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682560421748"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560421748"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560421748"}]},"ts":"1682560421748"} 2023-04-27 01:53:41,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 8b6eb92cd0f6859befa2a3fe078eb91a, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:53:41,917 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:53:41,918 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8b6eb92cd0f6859befa2a3fe078eb91a, NAME => 'hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:53:41,919 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:41,919 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:41,919 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:41,919 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:41,922 INFO [StoreOpener-8b6eb92cd0f6859befa2a3fe078eb91a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:41,925 DEBUG [StoreOpener-8b6eb92cd0f6859befa2a3fe078eb91a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/info 2023-04-27 01:53:41,925 DEBUG [StoreOpener-8b6eb92cd0f6859befa2a3fe078eb91a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/info 2023-04-27 01:53:41,925 INFO [StoreOpener-8b6eb92cd0f6859befa2a3fe078eb91a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8b6eb92cd0f6859befa2a3fe078eb91a columnFamilyName info 2023-04-27 01:53:41,926 INFO [StoreOpener-8b6eb92cd0f6859befa2a3fe078eb91a-1] regionserver.HStore(310): Store=8b6eb92cd0f6859befa2a3fe078eb91a/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:41,929 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:41,932 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:41,938 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:53:41,942 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:53:41,943 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8b6eb92cd0f6859befa2a3fe078eb91a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75282231, jitterRate=0.12179265916347504}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-27 01:53:41,943 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8b6eb92cd0f6859befa2a3fe078eb91a: 2023-04-27 01:53:41,945 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a., pid=6, masterSystemTime=1682560421910 2023-04-27 01:53:41,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:53:41,950 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:53:41,952 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=8b6eb92cd0f6859befa2a3fe078eb91a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:41,953 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682560421951"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560421951"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560421951"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560421951"}]},"ts":"1682560421951"} 2023-04-27 01:53:41,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-27 01:53:41,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 8b6eb92cd0f6859befa2a3fe078eb91a, server=jenkins-hbase4.apache.org,38085,1682560417985 in 202 msec 2023-04-27 01:53:41,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-27 01:53:41,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=8b6eb92cd0f6859befa2a3fe078eb91a, ASSIGN in 369 msec 2023-04-27 01:53:41,967 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 01:53:41,967 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560421967"}]},"ts":"1682560421967"} 2023-04-27 01:53:41,970 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-27 01:53:41,974 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 01:53:41,977 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3190 sec 2023-04-27 01:53:42,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-27 01:53:42,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-27 01:53:42,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:53:42,104 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-27 01:53:42,122 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-27 01:53:42,129 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-04-27 01:53:42,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-27 01:53:42,153 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-27 01:53:42,158 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-04-27 01:53:42,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-27 01:53:42,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-27 01:53:42,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.114sec 2023-04-27 01:53:42,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-27 01:53:42,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-27 01:53:42,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-27 01:53:42,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34643,1682560416787-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-27 01:53:42,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34643,1682560416787-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-27 01:53:42,190 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-27 01:53:42,229 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x5c01f436 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:42,234 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461317a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:42,250 DEBUG [hconnection-0x230c67ce-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:42,262 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:42,279 INFO [Listener at localhost/42495] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:53:42,303 INFO [Listener at localhost/42495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=333, ProcessCount=170, AvailableMemoryMB=12717 2023-04-27 01:53:42,319 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 01:53:42,322 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 01:53:42,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:53:42,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:53:42,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 01:53:42,339 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 01:53:42,342 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 01:53:42,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-27 01:53:42,345 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:42,346 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905 empty. 2023-04-27 01:53:42,347 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:42,347 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:53:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 01:53:42,363 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:53:42,365 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4931dc2885b7c715265d908d4b978905, NAME => 'TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp 2023-04-27 01:53:42,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 01:53:42,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 01:53:42,777 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:42,777 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4931dc2885b7c715265d908d4b978905, disabling compactions & flushes 2023-04-27 01:53:42,778 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:42,778 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:42,778 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. after waiting 0 ms 2023-04-27 01:53:42,778 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:42,778 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:42,778 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:42,782 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 01:53:42,783 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560422783"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560422783"}]},"ts":"1682560422783"} 2023-04-27 01:53:42,786 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 01:53:42,788 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 01:53:42,788 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560422788"}]},"ts":"1682560422788"} 2023-04-27 01:53:42,790 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 01:53:42,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4931dc2885b7c715265d908d4b978905, ASSIGN}] 2023-04-27 01:53:42,796 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4931dc2885b7c715265d908d4b978905, ASSIGN 2023-04-27 01:53:42,797 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4931dc2885b7c715265d908d4b978905, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=false 2023-04-27 01:53:42,949 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4931dc2885b7c715265d908d4b978905, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:42,950 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560422949"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560422949"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560422949"}]},"ts":"1682560422949"} 2023-04-27 01:53:42,953 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:53:42,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 01:53:43,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:43,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4931dc2885b7c715265d908d4b978905, NAME => 'TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:53:43,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:53:43,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,117 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,119 DEBUG [StoreOpener-4931dc2885b7c715265d908d4b978905-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A 2023-04-27 01:53:43,119 DEBUG [StoreOpener-4931dc2885b7c715265d908d4b978905-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A 2023-04-27 01:53:43,121 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:53:43,121 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4931dc2885b7c715265d908d4b978905 columnFamilyName A 2023-04-27 01:53:43,122 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.HStore(310): Store=4931dc2885b7c715265d908d4b978905/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:43,123 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,125 DEBUG [StoreOpener-4931dc2885b7c715265d908d4b978905-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B 2023-04-27 01:53:43,125 DEBUG [StoreOpener-4931dc2885b7c715265d908d4b978905-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B 2023-04-27 01:53:43,126 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:53:43,126 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4931dc2885b7c715265d908d4b978905 columnFamilyName B 2023-04-27 01:53:43,127 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.HStore(310): Store=4931dc2885b7c715265d908d4b978905/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:43,127 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,129 DEBUG [StoreOpener-4931dc2885b7c715265d908d4b978905-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C 2023-04-27 01:53:43,129 DEBUG [StoreOpener-4931dc2885b7c715265d908d4b978905-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C 2023-04-27 01:53:43,129 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:53:43,130 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4931dc2885b7c715265d908d4b978905 columnFamilyName C 2023-04-27 01:53:43,130 INFO [StoreOpener-4931dc2885b7c715265d908d4b978905-1] regionserver.HStore(310): Store=4931dc2885b7c715265d908d4b978905/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:53:43,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:43,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:53:43,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:53:43,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4931dc2885b7c715265d908d4b978905; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65391432, jitterRate=-0.025591731071472168}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:53:43,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:43,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., pid=11, masterSystemTime=1682560423106 2023-04-27 01:53:43,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:43,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:43,152 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4931dc2885b7c715265d908d4b978905, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:43,152 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560423152"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560423152"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560423152"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560423152"}]},"ts":"1682560423152"} 2023-04-27 01:53:43,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-27 01:53:43,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 in 203 msec 2023-04-27 01:53:43,163 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-27 01:53:43,163 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4931dc2885b7c715265d908d4b978905, ASSIGN in 365 msec 2023-04-27 01:53:43,164 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 01:53:43,165 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560423164"}]},"ts":"1682560423164"} 2023-04-27 01:53:43,167 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 01:53:43,171 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 01:53:43,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 837 msec 2023-04-27 01:53:43,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 01:53:43,468 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-27 01:53:43,473 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x36d9637e to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,477 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@348b992a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,480 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,483 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,486 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 01:53:43,495 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 01:53:43,502 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x6b41e306 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,509 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2732b8c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,510 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x2411a862 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,514 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72ea4a65, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,514 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x377933e4 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,518 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1645b6b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,518 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x3e8196e0 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,522 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b52aa42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,522 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x1ca4a062 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,527 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1710407d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,527 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x75f5a5e6 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,531 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ef1151, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,532 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x71f9f426 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,567 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29fba56a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,568 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x5ae6c42b to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,572 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ec2fae8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,573 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x726fb449 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:53:43,578 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e44f1b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:53:43,589 DEBUG [hconnection-0x2a81f259-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,593 DEBUG [hconnection-0x5b5f7a77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,593 DEBUG [hconnection-0x527b0319-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,593 DEBUG [hconnection-0x544eb923-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,600 DEBUG [hconnection-0x1bd47f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,600 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,601 DEBUG [hconnection-0x7a95e8d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,601 DEBUG [hconnection-0x6ff4c9b4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,602 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41130, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,609 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,615 DEBUG [hconnection-0x7d721dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,624 DEBUG [hconnection-0x1e045d2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:53:43,626 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:43,634 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,644 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,646 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41188, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,651 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:43,656 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:43,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:43,657 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41206, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:53:43,661 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:43,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:43,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:43,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:43,681 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:43,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:43,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:43,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:43,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:43,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:43,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:43,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:43,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:43,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:43,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:43,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:43,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:43,696 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:43,698 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 01:53:43,713 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:43,714 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:43,715 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:43,715 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:43,715 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:43,715 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:43,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:43,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:43,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682560423848, max timestamp=1682560423886Num uniques 859; 2023-04-27 01:53:43,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-27 01:53:43,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682560423848, max timestamp=1682560423886Num uniques 859; 2023-04-27 01:53:43,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-27 01:53:43,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682560423848, max timestamp=1682560423886Num uniques 859; 2023-04-27 01:53:43,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-27 01:53:43,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682560423961, max timestamp=1682560423991Num uniques 1009; 2023-04-27 01:53:43,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-27 01:53:44,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560423961, max timestamp=1682560423991Num uniques 1009; 2023-04-27 01:53:44,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-27 01:53:44,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560423961, max timestamp=1682560423991Num uniques 1009; 2023-04-27 01:53:44,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-27 01:53:44,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560484042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682560424030, max timestamp=1682560424044Num uniques 1027; 2023-04-27 01:53:44,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 01:53:44,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682560424030, max timestamp=1682560424042Num uniques 1027; 2023-04-27 01:53:44,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-27 01:53:44,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682560424030, max timestamp=1682560424042Num uniques 1127; 2023-04-27 01:53:44,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51603, new segment data size=6870, suffix heap size=145176,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=150 2023-04-27 01:53:44,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682560424084, max timestamp=1682560424101Num uniques 1009; 2023-04-27 01:53:44,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-27 01:53:44,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682560424084, max timestamp=1682560424101Num uniques 1036; 2023-04-27 01:53:44,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-27 01:53:44,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682560424084, max timestamp=1682560424101Num uniques 1009; 2023-04-27 01:53:44,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46223, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-27 01:53:44,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11212, totalHeapSize=25720, min timestamp=1682560424222, max timestamp=1682560424231Num uniques 1095; 2023-04-27 01:53:44,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50142, new segment data size=11212, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-27 01:53:44,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682560424222, max timestamp=1682560424231Num uniques 1095; 2023-04-27 01:53:44,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682560424222, max timestamp=1682560424231Num uniques 1095; 2023-04-27 01:53:44,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-27 01:53:44,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-27 01:53:44,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560484268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560484268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560484274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560484271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560484282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,304 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/fbdbca1527ec45d0a9e63af4931d80c6 2023-04-27 01:53:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560484386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560484385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560484388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560484397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560484400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,422 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/2c2f762b237c48bd8c7474563c40c104 2023-04-27 01:53:44,531 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/04671756a40749cea17a7e3f3dc6c504 2023-04-27 01:53:44,547 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/fbdbca1527ec45d0a9e63af4931d80c6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/fbdbca1527ec45d0a9e63af4931d80c6 2023-04-27 01:53:44,559 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/fbdbca1527ec45d0a9e63af4931d80c6, entries=150, sequenceid=9, filesize=11.6 K 2023-04-27 01:53:44,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/2c2f762b237c48bd8c7474563c40c104 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2c2f762b237c48bd8c7474563c40c104 2023-04-27 01:53:44,595 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2c2f762b237c48bd8c7474563c40c104, entries=150, sequenceid=9, filesize=11.6 K 2023-04-27 01:53:44,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/04671756a40749cea17a7e3f3dc6c504 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/04671756a40749cea17a7e3f3dc6c504 2023-04-27 01:53:44,623 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/04671756a40749cea17a7e3f3dc6c504, entries=150, sequenceid=9, filesize=11.6 K 2023-04-27 01:53:44,633 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 4931dc2885b7c715265d908d4b978905 in 933ms, sequenceid=9, compaction requested=false 2023-04-27 01:53:44,635 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-27 01:53:44,638 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:44,638 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:44,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,645 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=154.31 KB heapSize=437.39 KB 2023-04-27 01:53:44,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:44,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 01:53:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:44,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:44,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:44,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:44,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 01:53:44,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:44,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 01:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77457bb2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:44,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:44,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,660 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:44,666 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:44,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:44,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:44,669 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:44,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:44,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:44,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:44,669 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:44,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560484655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:44,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560484655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560484669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560484670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560484676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:44,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:44,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:44,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:44,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:44,695 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:44,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:44,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:44,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:44,703 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:44,703 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:44,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560484778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560484780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560484789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560484790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,803 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:44,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:44,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:44,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:44,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560484795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:44,836 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:44,836 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:44,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:44,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:44,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:44,844 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:44,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:44,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:44,845 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:44,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:44,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:44,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:44,874 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:44,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:44,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:44,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:44,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:44,876 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:44,876 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,876 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:44,876 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:44,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:44,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:44,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c003ff[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:44,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:44,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:44,887 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:44,888 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,888 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:44,896 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:44,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:44,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:44,917 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:44,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:44,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:44,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:44,917 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:44,923 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,923 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:44,923 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:44,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:44,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:44,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:44,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:44,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:44,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:44,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:44,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:44,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:44,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:44,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:44,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:44,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:44,942 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:44,942 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:45,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560484986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560484986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560485003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560485004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560485026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,042 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:45,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:45,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:45,081 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:45,081 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:45,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:45,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:45,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,091 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:45,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:45,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:45,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:45,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:45,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:45,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:45,105 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:45,111 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:45,111 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:45,113 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:45,114 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:45,114 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:45,114 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:45,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/95a5a49dfa904007bfd3652b63a9b0b0 2023-04-27 01:53:45,124 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:45,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:45,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:45,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bcb58d1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:45,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:45,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:45,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:45,131 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:45,131 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:45,144 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:45,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:45,152 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:45,152 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:45,152 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:45,152 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:45,152 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:45,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:45,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:45,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,157 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:45,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:45,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:45,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:45,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:45,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:45,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,176 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:45,176 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:45,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:45,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/351e1f1c319845459a5a3d2a9a97dd1e 2023-04-27 01:53:45,277 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:45,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:45,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:45,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560485308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560485310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560485310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560485310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:45,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560485333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:45,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:45,372 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:45,372 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:45,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:45,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:45,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/854b14cf67c24a47a199526c60fc71a0 2023-04-27 01:53:45,391 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,392 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,393 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:45,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:45,394 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,394 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:45,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:45,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:45,403 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:45,403 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:45,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/95a5a49dfa904007bfd3652b63a9b0b0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/95a5a49dfa904007bfd3652b63a9b0b0 2023-04-27 01:53:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:45,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:45,428 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:45,428 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:45,428 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:45,428 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:45,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:45,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:45,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:45,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:45,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:45,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/95a5a49dfa904007bfd3652b63a9b0b0, entries=200, sequenceid=116, filesize=13.9 K 2023-04-27 01:53:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:45,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@795b291[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:45,443 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:45,443 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/351e1f1c319845459a5a3d2a9a97dd1e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351e1f1c319845459a5a3d2a9a97dd1e 2023-04-27 01:53:45,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:45,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:45,453 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:45,453 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:45,453 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:45,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:45,461 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:45,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:45,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:45,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:45,461 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:45,463 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,463 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,463 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:45,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,494 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,494 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:45,494 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:45,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:45,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:45,495 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:45,495 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:45,495 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:45,495 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,495 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:45,495 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,496 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:45,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,495 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:45,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:45,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351e1f1c319845459a5a3d2a9a97dd1e, entries=150, sequenceid=116, filesize=11.6 K 2023-04-27 01:53:45,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/854b14cf67c24a47a199526c60fc71a0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/854b14cf67c24a47a199526c60fc71a0 2023-04-27 01:53:45,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/854b14cf67c24a47a199526c60fc71a0, entries=150, sequenceid=116, filesize=11.6 K 2023-04-27 01:53:45,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~437.34 KB/447840, currentSize=26.84 KB/27480 for 4931dc2885b7c715265d908d4b978905 in 888ms, sequenceid=116, compaction requested=false 2023-04-27 01:53:45,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:45,597 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:45,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:45,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:45,660 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:45,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:45,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:45,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:45,674 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,674 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,675 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:45,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:45,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:45,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:45,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:45,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:45,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:45,694 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:45,694 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:45,694 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 01:53:45,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:45,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:45,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:45,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:45,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:45,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:45,835 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/3221cf3739e24f4db7e4657ad341b000 2023-04-27 01:53:45,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:45,921 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/17da083b0a9e4899be8ab894bdf4fe76 2023-04-27 01:53:45,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682560425877, max timestamp=1682560425890Num uniques 859; 2023-04-27 01:53:45,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9556, totalHeapSize=21976, min timestamp=1682560425877, max timestamp=1682560425890Num uniques 859; 2023-04-27 01:53:45,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:45,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39326, new segment data size=9556, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:45,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9558, totalHeapSize=21976, min timestamp=1682560425877, max timestamp=1682560425890Num uniques 859; 2023-04-27 01:53:45,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39328, new segment data size=9558, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:45,976 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/d99ab51cd7fd467bb97291fa3e40737a 2023-04-27 01:53:45,989 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/3221cf3739e24f4db7e4657ad341b000 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3221cf3739e24f4db7e4657ad341b000 2023-04-27 01:53:46,002 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3221cf3739e24f4db7e4657ad341b000, entries=150, sequenceid=123, filesize=11.6 K 2023-04-27 01:53:46,017 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/17da083b0a9e4899be8ab894bdf4fe76 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/17da083b0a9e4899be8ab894bdf4fe76 2023-04-27 01:53:46,034 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/17da083b0a9e4899be8ab894bdf4fe76, entries=150, sequenceid=123, filesize=11.6 K 2023-04-27 01:53:46,036 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/d99ab51cd7fd467bb97291fa3e40737a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d99ab51cd7fd467bb97291fa3e40737a 2023-04-27 01:53:46,054 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d99ab51cd7fd467bb97291fa3e40737a, entries=150, sequenceid=123, filesize=11.6 K 2023-04-27 01:53:46,058 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=140.89 KB/144270 for 4931dc2885b7c715265d908d4b978905 in 364ms, sequenceid=123, compaction requested=true 2023-04-27 01:53:46,059 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:46,059 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:46,059 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:46,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426000, max timestamp=1682560426056Num uniques 1068; 2023-04-27 01:53:46,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-27 01:53:46,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426000, max timestamp=1682560426056Num uniques 1068; 2023-04-27 01:53:46,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-27 01:53:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:46,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-27 01:53:46,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:46,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426000, max timestamp=1682560426056Num uniques 1068; 2023-04-27 01:53:46,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-27 01:53:46,088 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 01:53:46,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:46,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:46,096 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:46,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:46,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:46,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:46,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:46,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b51c13a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:46,100 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:46,100 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,102 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,102 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:46,106 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,106 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:46,109 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,109 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:46,109 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:46,109 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:46,113 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:46,114 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:46,114 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:46,114 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:46,114 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:46,116 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,116 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,116 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,117 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,119 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:46,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:46,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:46,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:46,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:46,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:46,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:46,144 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,145 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:46,145 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:46,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:46,246 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:46,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:46,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/f78c73d14c4f406b9b5bab2d1423e3a3 2023-04-27 01:53:46,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:46,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560426236, max timestamp=1682560426265Num uniques 859; 2023-04-27 01:53:46,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560426236, max timestamp=1682560426265Num uniques 859; 2023-04-27 01:53:46,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:46,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:46,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560426236, max timestamp=1682560426265Num uniques 859; 2023-04-27 01:53:46,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:46,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:46,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:46,276 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:46,276 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:46,286 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:46,286 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:46,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:46,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:46,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:46,291 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,291 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:46,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:46,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:46,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:46,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:46,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:46,306 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eebee54[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:46,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:46,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:46,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:46,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:46,316 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:46,325 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:46,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:46,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:46,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:46,325 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:46,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5 2023-04-27 01:53:46,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,327 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-27 01:53:46,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:46,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:46,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,329 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-27 01:53:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:46,330 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:46,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:46,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:46,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:46,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:46,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:46,342 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:46,342 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:46,341 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:46,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:46,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:46,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426346, max timestamp=1682560426362Num uniques 1018; 2023-04-27 01:53:46,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:46,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426346, max timestamp=1682560426365Num uniques 1018; 2023-04-27 01:53:46,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:46,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426346, max timestamp=1682560426365Num uniques 1018; 2023-04-27 01:53:46,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:46,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/1cdbcf8a6e384c4cac4553c65dfe0d20 2023-04-27 01:53:46,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682560426398, max timestamp=1682560426403Num uniques 1027; 2023-04-27 01:53:46,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 01:53:46,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682560426398, max timestamp=1682560426403Num uniques 1027; 2023-04-27 01:53:46,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 01:53:46,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682560426398, max timestamp=1682560426403Num uniques 1027; 2023-04-27 01:53:46,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 01:53:46,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/f78c73d14c4f406b9b5bab2d1423e3a3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/f78c73d14c4f406b9b5bab2d1423e3a3 2023-04-27 01:53:46,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/f78c73d14c4f406b9b5bab2d1423e3a3, entries=200, sequenceid=161, filesize=14.1 K 2023-04-27 01:53:46,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5 2023-04-27 01:53:46,442 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:46,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:46,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5, entries=150, sequenceid=161, filesize=11.8 K 2023-04-27 01:53:46,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/1cdbcf8a6e384c4cac4553c65dfe0d20 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1cdbcf8a6e384c4cac4553c65dfe0d20 2023-04-27 01:53:46,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:46,460 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:46,460 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:46,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:46,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:46,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,464 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:46,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:46,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:46,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:46,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:46,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:46,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:46,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:46,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:46,469 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:46,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1cdbcf8a6e384c4cac4553c65dfe0d20, entries=150, sequenceid=161, filesize=11.8 K 2023-04-27 01:53:46,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=134.18 KB/137400 for 4931dc2885b7c715265d908d4b978905 in 410ms, sequenceid=161, compaction requested=true 2023-04-27 01:53:46,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:46,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682560426449, max timestamp=1682560426478Num uniques 1036; 2023-04-27 01:53:46,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-27 01:53:46,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682560426449, max timestamp=1682560426478Num uniques 1036; 2023-04-27 01:53:46,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:46,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-27 01:53:46,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682560426449, max timestamp=1682560426478Num uniques 1036; 2023-04-27 01:53:46,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-27 01:53:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:46,492 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:46,492 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:46,492 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=53.67 KB heapSize=134.02 KB 2023-04-27 01:53:46,493 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:46,493 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:46,493 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:46,493 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:46,493 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:46,493 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:46,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:46,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:46,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:46,503 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:46,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:46,503 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:46,516 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:46,518 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:46,519 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52424 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:46,519 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:46,519 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:46,519 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:46,519 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2c2f762b237c48bd8c7474563c40c104, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351e1f1c319845459a5a3d2a9a97dd1e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/17da083b0a9e4899be8ab894bdf4fe76, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=46.6 K 2023-04-27 01:53:46,519 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/fbdbca1527ec45d0a9e63af4931d80c6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/95a5a49dfa904007bfd3652b63a9b0b0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3221cf3739e24f4db7e4657ad341b000, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/f78c73d14c4f406b9b5bab2d1423e3a3] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=51.2 K 2023-04-27 01:53:46,522 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting fbdbca1527ec45d0a9e63af4931d80c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682560423665 2023-04-27 01:53:46,523 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2c2f762b237c48bd8c7474563c40c104, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682560423665 2023-04-27 01:53:46,525 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 95a5a49dfa904007bfd3652b63a9b0b0, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682560424269 2023-04-27 01:53:46,536 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 351e1f1c319845459a5a3d2a9a97dd1e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682560424269 2023-04-27 01:53:46,537 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 17da083b0a9e4899be8ab894bdf4fe76, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682560424654 2023-04-27 01:53:46,537 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3221cf3739e24f4db7e4657ad341b000, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682560424654 2023-04-27 01:53:46,538 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f78c73d14c4f406b9b5bab2d1423e3a3, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682560426000 2023-04-27 01:53:46,539 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2ae8cbe45f5a46e4a0c47bdc23b0edc5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682560426006 2023-04-27 01:53:46,554 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/395b29307238497ba7945ede954d18f8 2023-04-27 01:53:46,586 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#13 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:46,593 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#14 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:46,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560426577, max timestamp=1682560426590Num uniques 859; 2023-04-27 01:53:46,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:46,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560426577, max timestamp=1682560426590Num uniques 859; 2023-04-27 01:53:46,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:46,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560426577, max timestamp=1682560426590Num uniques 859; 2023-04-27 01:53:46,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:46,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560486657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560486666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,678 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/163ccd0894464c2baafa5eaa64edd889 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/163ccd0894464c2baafa5eaa64edd889 2023-04-27 01:53:46,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560486669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560486669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560486677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,703 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 163ccd0894464c2baafa5eaa64edd889(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:46,703 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:46,703 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=12, startTime=1682560426481; duration=0sec 2023-04-27 01:53:46,704 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:46,704 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:46,707 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:46,708 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:46,708 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:46,708 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/04671756a40749cea17a7e3f3dc6c504, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/854b14cf67c24a47a199526c60fc71a0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d99ab51cd7fd467bb97291fa3e40737a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1cdbcf8a6e384c4cac4553c65dfe0d20] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=46.6 K 2023-04-27 01:53:46,709 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 04671756a40749cea17a7e3f3dc6c504, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682560423665 2023-04-27 01:53:46,710 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 854b14cf67c24a47a199526c60fc71a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682560424269 2023-04-27 01:53:46,710 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d99ab51cd7fd467bb97291fa3e40737a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682560424654 2023-04-27 01:53:46,711 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 1cdbcf8a6e384c4cac4553c65dfe0d20, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682560426006 2023-04-27 01:53:46,748 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#16 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:46,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560486783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560486782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560486784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,795 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/9f68edd2ea9e4a6e880f9ccb9b9dd151 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/9f68edd2ea9e4a6e880f9ccb9b9dd151 2023-04-27 01:53:46,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560486784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560486788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,819 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 9f68edd2ea9e4a6e880f9ccb9b9dd151(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:46,820 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:46,820 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=12, startTime=1682560426503; duration=0sec 2023-04-27 01:53:46,820 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:46,959 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-27 01:53:46,960 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-27 01:53:46,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560486992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560486992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:46,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:46,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560486992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560486998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560487008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,066 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/480288958a4d428faaed3decccd7ded8 2023-04-27 01:53:47,076 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/b18a1ef7375e4c86857edb7ebe2ac5e2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b18a1ef7375e4c86857edb7ebe2ac5e2 2023-04-27 01:53:47,093 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into b18a1ef7375e4c86857edb7ebe2ac5e2(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:47,094 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:47,094 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=12, startTime=1682560426503; duration=0sec 2023-04-27 01:53:47,094 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:47,110 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/a1da99a232f14e579d8e458fb3240727 2023-04-27 01:53:47,121 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/395b29307238497ba7945ede954d18f8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/395b29307238497ba7945ede954d18f8 2023-04-27 01:53:47,135 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/395b29307238497ba7945ede954d18f8, entries=400, sequenceid=235, filesize=23.4 K 2023-04-27 01:53:47,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/480288958a4d428faaed3decccd7ded8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/480288958a4d428faaed3decccd7ded8 2023-04-27 01:53:47,152 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/480288958a4d428faaed3decccd7ded8, entries=150, sequenceid=235, filesize=11.8 K 2023-04-27 01:53:47,157 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/a1da99a232f14e579d8e458fb3240727 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1da99a232f14e579d8e458fb3240727 2023-04-27 01:53:47,175 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1da99a232f14e579d8e458fb3240727, entries=150, sequenceid=235, filesize=11.8 K 2023-04-27 01:53:47,178 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~133.97 KB/137184, currentSize=134.18 KB/137400 for 4931dc2885b7c715265d908d4b978905 in 685ms, sequenceid=235, compaction requested=false 2023-04-27 01:53:47,179 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:47,179 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:47,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd73488[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:47,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:47,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:47,204 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,207 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:47,209 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,209 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:47,209 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:47,209 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:47,210 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:47,210 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:47,210 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:47,210 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:47,212 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,212 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,212 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,214 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,219 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,224 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,225 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,226 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,226 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:47,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:47,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,233 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:47,233 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:47,233 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:47,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426631, max timestamp=1682560427300Num uniques 1018; 2023-04-27 01:53:47,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:47,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=102.87 KB heapSize=292.20 KB 2023-04-27 01:53:47,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:47,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426631, max timestamp=1682560427300Num uniques 1018; 2023-04-27 01:53:47,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:47,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682560426631, max timestamp=1682560427300Num uniques 1018; 2023-04-27 01:53:47,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:47,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:47,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:47,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:47,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:47,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:47,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:47,333 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:47,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:47,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:47,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:47,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:47,348 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:47,348 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:47,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:47,350 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,350 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,351 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,351 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,353 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,353 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:47,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:47,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:47,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:47,359 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,360 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,361 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,361 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257c0ed9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:47,363 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:47,363 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,364 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:47,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:47,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:47,366 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:47,370 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:47,370 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:47,371 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:47,371 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:47,371 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:47,372 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,372 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,372 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,373 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,375 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:47,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:47,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,381 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:47,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,383 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:47,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,383 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:47,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560427342, max timestamp=1682560427399Num uniques 859; 2023-04-27 01:53:47,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560427342, max timestamp=1682560427399Num uniques 859; 2023-04-27 01:53:47,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:47,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560427342, max timestamp=1682560427399Num uniques 859; 2023-04-27 01:53:47,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:47,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:47,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7684, totalHeapSize=17712, min timestamp=1682560427442, max timestamp=1682560427453Num uniques 1018; 2023-04-27 01:53:47,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46614, new segment data size=7684, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:47,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682560427435, max timestamp=1682560427453Num uniques 1018; 2023-04-27 01:53:47,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-27 01:53:47,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7685, totalHeapSize=17712, min timestamp=1682560427442, max timestamp=1682560427453Num uniques 1018; 2023-04-27 01:53:47,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46615, new segment data size=7685, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 01:53:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682560427467, max timestamp=1682560427478Num uniques 1027; 2023-04-27 01:53:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-27 01:53:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682560427467, max timestamp=1682560427478Num uniques 1127; 2023-04-27 01:53:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-27 01:53:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682560427467, max timestamp=1682560427478Num uniques 1027; 2023-04-27 01:53:47,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-27 01:53:47,483 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:47,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:47,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:47,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:47,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:47,490 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:47,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,491 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:47,491 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:47,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,494 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,494 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,495 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,496 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:47,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:47,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:47,503 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,504 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:47,504 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:47,504 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:47,506 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,507 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,507 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,507 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:47,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f1535d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:47,511 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:47,511 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,512 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:47,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:47,516 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:47,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:47,525 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:47,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:47,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:47,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:47,525 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,527 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560487515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560487517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,527 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,530 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560487515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560487523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560487523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,537 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,543 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,543 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,543 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,543 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:47,543 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,545 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,545 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:47,545 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,545 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,545 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,545 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:47,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:47,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560487630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560487630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560487636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560487637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560487638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,645 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:47,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:47,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:47,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:47,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:47,654 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:47,654 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:47,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:47,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,658 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,658 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,658 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:47,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:47,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:47,664 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,664 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:47,664 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:47,664 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:47,667 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:47,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11098a59[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:47,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:47,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,671 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:47,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:47,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:47,672 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,672 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:47,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:47,691 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:47,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:47,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:47,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:47,691 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:47,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,695 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:47,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,705 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:47,705 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:47,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/16de9f877caa4463afc1a48d6dbedc9c 2023-04-27 01:53:47,805 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:47,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:47,811 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:47,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:47,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:47,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:47,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:47,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:47,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:47,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:47,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:47,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:47,817 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,817 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:47,818 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:47,818 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:47,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:47,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27cc8f84[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:47,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:47,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:47,831 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,831 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:47,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,834 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:47,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:47,839 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:47,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:47,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:47,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:47,839 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:47,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560487836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560487837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560487838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,843 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560487843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,845 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560487844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,850 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,850 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,850 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:47,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,850 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,852 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:47,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,852 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:47,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:47,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:47,952 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:47,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:47,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:47,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:47,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:47,962 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:47,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,963 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:47,963 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:47,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:47,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:47,966 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:47,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:47,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:47,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:47,970 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,970 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:47,970 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:47,971 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,976 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,976 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62835762[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:47,980 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:47,980 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,981 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:47,982 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,982 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:47,983 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,984 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:47,984 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:47,984 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:47,990 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:47,990 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:47,990 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:47,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:47,991 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:47,999 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,999 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:47,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,000 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:48,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:48,001 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,005 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:48,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:48,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:48,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,008 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:48,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:48,009 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:48,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:48,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:48,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:48,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:48,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:48,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:48,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:48,119 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:48,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:48,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:48,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:48,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,124 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:48,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:48,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,125 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,125 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:48,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:48,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:48,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:48,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:48,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:48,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:48,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:48,144 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,145 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:48,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560488142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18129bd4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:48,149 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:48,149 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,151 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:48,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560488146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560488144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:48,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560488153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560488153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,156 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:48,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:48,161 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:48,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:48,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:48,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:48,161 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:48,163 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,163 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,163 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:48,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:48,164 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:48,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,166 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:48,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:48,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:48,172 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:48,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:48,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:48,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,173 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:48,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:48,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:48,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:48,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/dea4d02b46234615800c74d6a335a04b 2023-04-27 01:53:48,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/a1a6f8357e004207981e6b1ff7b24154 2023-04-27 01:53:48,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/16de9f877caa4463afc1a48d6dbedc9c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/16de9f877caa4463afc1a48d6dbedc9c 2023-04-27 01:53:48,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:48,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/16de9f877caa4463afc1a48d6dbedc9c, entries=200, sequenceid=277, filesize=14.3 K 2023-04-27 01:53:48,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:48,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/dea4d02b46234615800c74d6a335a04b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/dea4d02b46234615800c74d6a335a04b 2023-04-27 01:53:48,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:48,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:48,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:48,282 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:48,282 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:48,283 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:48,283 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:48,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,287 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:48,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:48,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/dea4d02b46234615800c74d6a335a04b, entries=150, sequenceid=277, filesize=11.9 K 2023-04-27 01:53:48,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/a1a6f8357e004207981e6b1ff7b24154 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1a6f8357e004207981e6b1ff7b24154 2023-04-27 01:53:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:48,297 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,297 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:48,297 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:48,297 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:48,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,300 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@451481cd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:48,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:48,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:48,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:48,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:48,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:48,312 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:48,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1a6f8357e004207981e6b1ff7b24154, entries=150, sequenceid=277, filesize=11.9 K 2023-04-27 01:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:48,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=161.02 KB/164880 for 4931dc2885b7c715265d908d4b978905 in 1015ms, sequenceid=277, compaction requested=true 2023-04-27 01:53:48,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:48,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:48,320 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:48,320 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:48,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:48,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,332 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 50790 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:48,333 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:48,333 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36396 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:48,333 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,333 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,333 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,333 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/163ccd0894464c2baafa5eaa64edd889, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/395b29307238497ba7945ede954d18f8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/16de9f877caa4463afc1a48d6dbedc9c] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=49.6 K 2023-04-27 01:53:48,333 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b18a1ef7375e4c86857edb7ebe2ac5e2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/480288958a4d428faaed3decccd7ded8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/dea4d02b46234615800c74d6a335a04b] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=35.5 K 2023-04-27 01:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:48,335 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:48,336 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:48,336 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:48,336 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:48,336 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:48,337 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 163ccd0894464c2baafa5eaa64edd889, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682560426006 2023-04-27 01:53:48,337 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b18a1ef7375e4c86857edb7ebe2ac5e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682560426006 2023-04-27 01:53:48,337 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 395b29307238497ba7945ede954d18f8, keycount=400, bloomtype=ROW, size=23.4 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682560426449 2023-04-27 01:53:48,338 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 480288958a4d428faaed3decccd7ded8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682560426449 2023-04-27 01:53:48,338 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 16de9f877caa4463afc1a48d6dbedc9c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560426631 2023-04-27 01:53:48,338 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting dea4d02b46234615800c74d6a335a04b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560426631 2023-04-27 01:53:48,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:48,342 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,345 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:48,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:48,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:48,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:48,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:48,350 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:48,350 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:48,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:48,351 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:48,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:48,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:48,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:48,356 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#21 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:48,361 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#22 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:48,450 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:48,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:48,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:48,463 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/e17804849541494b930227b7e66c20c5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e17804849541494b930227b7e66c20c5 2023-04-27 01:53:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:48,464 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:48,464 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:48,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:48,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:48,466 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/d3fc9aaf0969441a952810a323f92b8f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d3fc9aaf0969441a952810a323f92b8f 2023-04-27 01:53:48,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:48,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:48,472 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,473 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,473 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:48,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:48,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:48,478 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into e17804849541494b930227b7e66c20c5(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:48,478 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:48,479 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=13, startTime=1682560428320; duration=0sec 2023-04-27 01:53:48,479 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:48,479 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:48,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:48,481 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36396 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:48,481 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:48,481 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,481 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/9f68edd2ea9e4a6e880f9ccb9b9dd151, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1da99a232f14e579d8e458fb3240727, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1a6f8357e004207981e6b1ff7b24154] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=35.5 K 2023-04-27 01:53:48,482 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9f68edd2ea9e4a6e880f9ccb9b9dd151, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682560426006 2023-04-27 01:53:48,483 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a1da99a232f14e579d8e458fb3240727, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682560426449 2023-04-27 01:53:48,483 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a1a6f8357e004207981e6b1ff7b24154, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560426631 2023-04-27 01:53:48,488 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:48,491 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=161.02 KB heapSize=444.23 KB 2023-04-27 01:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:48,492 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:48,492 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:48,492 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:48,492 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:48,495 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into d3fc9aaf0969441a952810a323f92b8f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:48,495 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:48,495 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=13, startTime=1682560428320; duration=0sec 2023-04-27 01:53:48,496 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:48,505 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#24 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:48,549 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/17a3c918d9ff45daa630b02f46bc797b 2023-04-27 01:53:48,564 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/6923777d138a494ebaecf05d8b6c0ead as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6923777d138a494ebaecf05d8b6c0ead 2023-04-27 01:53:48,585 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 6923777d138a494ebaecf05d8b6c0ead(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:48,585 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:48,585 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=13, startTime=1682560428322; duration=0sec 2023-04-27 01:53:48,585 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:48,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:48,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:48,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560488661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560488669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560488670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560488670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560488670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560488771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560488777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560488778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560488778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560488778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560488979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560488981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560488982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560488982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:48,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:48,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560488984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,046 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/72eabe9719a54ebe99c98c8209a2d87b 2023-04-27 01:53:49,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560489283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41192 deadline: 1682560489285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41180 deadline: 1682560489286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560489288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560489290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,517 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/0279811906624ae19caf7f22fb279ec2 2023-04-27 01:53:49,530 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/17a3c918d9ff45daa630b02f46bc797b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/17a3c918d9ff45daa630b02f46bc797b 2023-04-27 01:53:49,548 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/17a3c918d9ff45daa630b02f46bc797b, entries=150, sequenceid=349, filesize=11.9 K 2023-04-27 01:53:49,552 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/72eabe9719a54ebe99c98c8209a2d87b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/72eabe9719a54ebe99c98c8209a2d87b 2023-04-27 01:53:49,580 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/72eabe9719a54ebe99c98c8209a2d87b, entries=150, sequenceid=349, filesize=11.9 K 2023-04-27 01:53:49,583 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/0279811906624ae19caf7f22fb279ec2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0279811906624ae19caf7f22fb279ec2 2023-04-27 01:53:49,597 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0279811906624ae19caf7f22fb279ec2, entries=150, sequenceid=349, filesize=11.9 K 2023-04-27 01:53:49,601 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~444.19 KB/454848, currentSize=33.54 KB/34350 for 4931dc2885b7c715265d908d4b978905 in 1110ms, sequenceid=349, compaction requested=false 2023-04-27 01:53:49,601 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:49,601 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:49,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,609 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6679644[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:49,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:49,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,620 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:49,620 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,620 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:49,620 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:49,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:49,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:49,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:49,622 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:49,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:49,630 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:49,630 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:49,630 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:49,631 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:49,631 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:49,633 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,633 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,633 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:49,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:49,634 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:49,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,642 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:49,642 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:49,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:49,642 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:49,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:49,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:49,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:49,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:49,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:49,643 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:49,643 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:49,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:49,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:49,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:49,743 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:49,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:49,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:49,752 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:49,752 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:49,754 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:49,754 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:49,755 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,755 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:49,757 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:49,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:49,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:49,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:49,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:49,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:49,769 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:49,769 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:49,770 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-27 01:53:49,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:49,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:49,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:49,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:49,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:49,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:49,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:49,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:49,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560429806, max timestamp=1682560429849Num uniques 859; 2023-04-27 01:53:49,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560429806, max timestamp=1682560429849Num uniques 859; 2023-04-27 01:53:49,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:49,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560429806, max timestamp=1682560429849Num uniques 859; 2023-04-27 01:53:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:49,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682560429869, max timestamp=1682560429875Num uniques 1068; 2023-04-27 01:53:49,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682560429869, max timestamp=1682560429875Num uniques 1068; 2023-04-27 01:53:49,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-27 01:53:49,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-27 01:53:49,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682560429869, max timestamp=1682560429875Num uniques 1068; 2023-04-27 01:53:49,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-27 01:53:49,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1682560429887, max timestamp=1682560429903Num uniques 1077; 2023-04-27 01:53:49,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=19552, suffix heap size=139976,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=427 2023-04-27 01:53:49,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682560429892, max timestamp=1682560429903Num uniques 1077; 2023-04-27 01:53:49,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-27 01:53:49,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682560429892, max timestamp=1682560429907Num uniques 1077; 2023-04-27 01:53:49,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-27 01:53:49,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682560429922, max timestamp=1682560429932Num uniques 1286; 2023-04-27 01:53:49,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=58896, new segment data size=13096, suffix heap size=161712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1286, new segment cells count=286 2023-04-27 01:53:49,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682560429922, max timestamp=1682560429932Num uniques 1036; 2023-04-27 01:53:49,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-27 01:53:49,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682560429922, max timestamp=1682560429932Num uniques 1036; 2023-04-27 01:53:49,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-27 01:53:49,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560489958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1682560489959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:49,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560489960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:49,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682560429956, max timestamp=1682560429962Num uniques 1145; 2023-04-27 01:53:49,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-27 01:53:49,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682560429956, max timestamp=1682560429962Num uniques 1145; 2023-04-27 01:53:49,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682560429956, max timestamp=1682560429962Num uniques 1145; 2023-04-27 01:53:49,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-27 01:53:49,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-27 01:53:50,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1682560429997, max timestamp=1682560430009Num uniques 1104; 2023-04-27 01:53:50,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-27 01:53:50,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1682560429997, max timestamp=1682560430009Num uniques 1104; 2023-04-27 01:53:50,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-27 01:53:50,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682560429997, max timestamp=1682560430009Num uniques 1104; 2023-04-27 01:53:50,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-27 01:53:50,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682560430036, max timestamp=1682560430065Num uniques 1063; 2023-04-27 01:53:50,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-27 01:53:50,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682560430036, max timestamp=1682560430065Num uniques 1063; 2023-04-27 01:53:50,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682560430036, max timestamp=1682560430065Num uniques 1063; 2023-04-27 01:53:50,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-27 01:53:50,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-27 01:53:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682560430081, max timestamp=1682560430093Num uniques 1022; 2023-04-27 01:53:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=17032, suffix heap size=134256,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=372 2023-04-27 01:53:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682560430081, max timestamp=1682560430093Num uniques 1022; 2023-04-27 01:53:50,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=17032, suffix heap size=134256,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=372 2023-04-27 01:53:50,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682560430081, max timestamp=1682560430093Num uniques 1022; 2023-04-27 01:53:50,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=17032, suffix heap size=134256,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=372 2023-04-27 01:53:50,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41206 deadline: 1682560490114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560490115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560490115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560490117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560490122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,215 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/5270dfe6db28434ab1d5f9fc42481038 2023-04-27 01:53:50,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560490221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560490223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560490223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560490225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560490226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,244 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/994107157dac47dcaec1ead8f81ac326 2023-04-27 01:53:50,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560490425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560490426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560490427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560490427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41128 deadline: 1682560490435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,671 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/3d5a46e09cdd438bb1f0ff650993189f 2023-04-27 01:53:50,679 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/5270dfe6db28434ab1d5f9fc42481038 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/5270dfe6db28434ab1d5f9fc42481038 2023-04-27 01:53:50,687 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/5270dfe6db28434ab1d5f9fc42481038, entries=150, sequenceid=359, filesize=11.9 K 2023-04-27 01:53:50,689 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/994107157dac47dcaec1ead8f81ac326 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/994107157dac47dcaec1ead8f81ac326 2023-04-27 01:53:50,697 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/994107157dac47dcaec1ead8f81ac326, entries=150, sequenceid=359, filesize=11.9 K 2023-04-27 01:53:50,699 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/3d5a46e09cdd438bb1f0ff650993189f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d5a46e09cdd438bb1f0ff650993189f 2023-04-27 01:53:50,708 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d5a46e09cdd438bb1f0ff650993189f, entries=150, sequenceid=359, filesize=11.9 K 2023-04-27 01:53:50,710 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 4931dc2885b7c715265d908d4b978905 in 941ms, sequenceid=359, compaction requested=true 2023-04-27 01:53:50,710 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:50,710 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:50,710 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:50,715 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,715 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:50,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:50,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea7f9ad[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:50,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:50,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,720 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:50,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:50,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:50,720 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,720 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:50,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:50,726 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:50,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:50,726 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:50,728 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,729 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,729 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:50,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:50,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:50,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:50,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=161.02 KB heapSize=444.70 KB 2023-04-27 01:53:50,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:50,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:50,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:50,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:50,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:50,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560490735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,739 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:50,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560490736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560490737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560490738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560490743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,748 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:50,748 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:50,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:50,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:50,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,749 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:50,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:50,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:50,751 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:50,751 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:50,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:50,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/e2eb0fe0cfa045c894f534ce8ee8ede7 2023-04-27 01:53:50,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/e80f4690e8894aa5a976c1f544253499 2023-04-27 01:53:50,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560490840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560490843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560490843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560490843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,851 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:50,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:50,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:50,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:50,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:50,888 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:50,888 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:50,893 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:50,893 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:50,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:50,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,896 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:50,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:50,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:50,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:50,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:50,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:50,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:50,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:50,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:50,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:50,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:50,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:50,925 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/1d84a6d56ce2481e8e3bd61bab10caf6 2023-04-27 01:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e70196[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:50,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:50,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:50,938 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,938 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:50,939 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:50,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/e2eb0fe0cfa045c894f534ce8ee8ede7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e2eb0fe0cfa045c894f534ce8ee8ede7 2023-04-27 01:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:50,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:50,948 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:50,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:50,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:50,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:50,948 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:50,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e2eb0fe0cfa045c894f534ce8ee8ede7, entries=200, sequenceid=516, filesize=14.3 K 2023-04-27 01:53:50,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:50,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/e80f4690e8894aa5a976c1f544253499 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/e80f4690e8894aa5a976c1f544253499 2023-04-27 01:53:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:50,958 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:50,966 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:50,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:50,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:50,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:50,971 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/e80f4690e8894aa5a976c1f544253499, entries=150, sequenceid=516, filesize=11.9 K 2023-04-27 01:53:50,973 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,973 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:50,973 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:50,973 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:50,973 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:50,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:50,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:50,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/1d84a6d56ce2481e8e3bd61bab10caf6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1d84a6d56ce2481e8e3bd61bab10caf6 2023-04-27 01:53:50,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:50,976 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:50,976 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:50,976 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:50,976 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:50,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1d84a6d56ce2481e8e3bd61bab10caf6, entries=150, sequenceid=516, filesize=11.9 K 2023-04-27 01:53:50,985 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~444.66 KB/455328, currentSize=26.84 KB/27480 for 4931dc2885b7c715265d908d4b978905 in 254ms, sequenceid=516, compaction requested=true 2023-04-27 01:53:50,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:50,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:50,985 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:50,985 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:50,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:50,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:50,988 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:50,989 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:50,989 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:50,989 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d3fc9aaf0969441a952810a323f92b8f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/72eabe9719a54ebe99c98c8209a2d87b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/994107157dac47dcaec1ead8f81ac326, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/e80f4690e8894aa5a976c1f544253499] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=47.8 K 2023-04-27 01:53:50,990 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51424 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:50,990 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:50,990 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:50,990 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e17804849541494b930227b7e66c20c5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/17a3c918d9ff45daa630b02f46bc797b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/5270dfe6db28434ab1d5f9fc42481038, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e2eb0fe0cfa045c894f534ce8ee8ede7] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=50.2 K 2023-04-27 01:53:50,990 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d3fc9aaf0969441a952810a323f92b8f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560426631 2023-04-27 01:53:50,992 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e17804849541494b930227b7e66c20c5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560426631 2023-04-27 01:53:50,992 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 72eabe9719a54ebe99c98c8209a2d87b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682560427505 2023-04-27 01:53:50,994 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 994107157dac47dcaec1ead8f81ac326, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682560428656 2023-04-27 01:53:50,994 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 17a3c918d9ff45daa630b02f46bc797b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682560427505 2023-04-27 01:53:50,994 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e80f4690e8894aa5a976c1f544253499, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682560430114 2023-04-27 01:53:50,995 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 5270dfe6db28434ab1d5f9fc42481038, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682560428656 2023-04-27 01:53:50,996 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e2eb0fe0cfa045c894f534ce8ee8ede7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682560430114 2023-04-27 01:53:51,010 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#33 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:51,011 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:51,038 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/85199c1ad627440ba207ed0d44123d65 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85199c1ad627440ba207ed0d44123d65 2023-04-27 01:53:51,048 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 85199c1ad627440ba207ed0d44123d65(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:51,049 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:51,049 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=12, startTime=1682560430985; duration=0sec 2023-04-27 01:53:51,049 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:51,049 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:51,057 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:51,057 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:51,057 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,057 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6923777d138a494ebaecf05d8b6c0ead, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0279811906624ae19caf7f22fb279ec2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d5a46e09cdd438bb1f0ff650993189f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1d84a6d56ce2481e8e3bd61bab10caf6] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=47.8 K 2023-04-27 01:53:51,057 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 6923777d138a494ebaecf05d8b6c0ead, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560426631 2023-04-27 01:53:51,058 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 0279811906624ae19caf7f22fb279ec2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682560427505 2023-04-27 01:53:51,058 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3d5a46e09cdd438bb1f0ff650993189f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1682560428656 2023-04-27 01:53:51,059 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 1d84a6d56ce2481e8e3bd61bab10caf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682560430114 2023-04-27 01:53:51,059 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/175b277e76a642dc9d37d5f864339d12 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/175b277e76a642dc9d37d5f864339d12 2023-04-27 01:53:51,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:51,066 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:53:51,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:51,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:51,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:51,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:51,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:51,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:51,070 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into 175b277e76a642dc9d37d5f864339d12(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:51,070 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:51,070 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=12, startTime=1682560430985; duration=0sec 2023-04-27 01:53:51,070 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:51,076 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:51,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:51,079 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:51,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:51,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:51,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:51,084 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:51,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:51,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:51,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,091 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:51,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:51,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:51,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,095 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:51,096 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:51,096 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,097 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:51,097 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:51,097 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:51,101 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,101 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,101 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,101 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@131cecee[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:51,104 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:51,104 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,109 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,110 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:51,110 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:51,110 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:51,110 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,110 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:51,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560491104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682560431086, max timestamp=1682560431109Num uniques 859; 2023-04-27 01:53:51,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 01:53:51,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682560431086, max timestamp=1682560431109Num uniques 859; 2023-04-27 01:53:51,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 01:53:51,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682560431086, max timestamp=1682560431109Num uniques 859; 2023-04-27 01:53:51,116 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 01:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:51,120 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:51,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:51,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:51,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:51,121 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:51,126 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,126 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,126 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,129 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,133 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,133 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,133 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:51,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:51,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,135 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:51,135 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:51,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560491132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560491135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560491139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560491215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,235 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:51,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:51,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560491242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560491243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560491243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:51,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:51,250 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:51,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,251 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:51,251 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:51,252 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,252 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560491252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,255 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,255 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,256 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,256 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:51,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:51,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:51,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:51,258 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,258 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:51,258 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:51,258 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,260 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:51,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29738d7f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:51,262 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:51,262 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:51,264 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,264 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:51,266 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:51,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:51,269 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:51,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:51,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:51,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:51,269 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:51,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,273 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,276 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,276 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:51,276 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:51,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:51,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,278 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:51,278 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:51,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,278 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,378 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:51,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:51,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:51,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:51,384 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:51,385 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:51,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:51,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,389 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:51,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:51,395 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,395 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:51,395 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:51,395 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:51,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:51,397 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:51,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6858636d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:51,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:51,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,402 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:51,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:51,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:51,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:51,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:51,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:51,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:51,410 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:51,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:51,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:51,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:51,410 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:51,413 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,413 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,413 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,414 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,417 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560491424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:51,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,436 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:51,436 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:51,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:51,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560491446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560491447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560491448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/2c1ed7dca392474884ce0a0a76aad176 2023-04-27 01:53:51,510 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/c24b5c3f428b49559a1b8460412cd2e5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c24b5c3f428b49559a1b8460412cd2e5 2023-04-27 01:53:51,518 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into c24b5c3f428b49559a1b8460412cd2e5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:51,518 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:51,518 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=12, startTime=1682560430986; duration=0sec 2023-04-27 01:53:51,518 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:51,536 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:51,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:51,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:51,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:51,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:51,541 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:51,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:51,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:51,542 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,542 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,543 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:51,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:51,546 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,546 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:51,546 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:51,546 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:51,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22792441[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:51,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,558 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:51,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:51,560 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:51,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:51,564 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:51,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:51,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:51,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:51,564 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:51,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,571 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,582 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:51,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:51,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:51,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:51,683 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:51,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:51,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:51,691 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:51,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:51,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:51,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:51,697 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,697 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:51,697 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:51,697 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:51,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:51,699 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,699 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,699 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,699 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f6c77c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:51,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:51,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,702 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:51,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:51,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:51,708 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:51,708 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:51,708 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:51,708 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:51,708 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:51,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,713 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:51,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:51,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:51,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,716 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:51,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:51,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560491728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560491752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560491754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560491754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:51,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:51,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:51,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:51,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:51,821 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:51,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:51,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:51,822 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,822 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:51,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:51,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:51,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:51,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:51,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:51,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:51,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,835 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63dbb0a8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:51,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:51,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,838 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:51,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:51,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:51,840 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:51,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:51,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:51,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:51,840 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:51,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,845 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,849 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,849 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,849 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,849 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,855 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,855 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,855 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,855 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,855 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:51,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:51,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:51,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,856 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:51,856 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:51,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/d146babeb67e4ca2af53225307df6244 2023-04-27 01:53:51,956 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:51,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:51,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:51,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:51,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:51,961 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:51,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:51,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:51,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:51,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:51,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:51,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:51,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:51,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:51,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:51,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:51,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:51,982 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,983 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,983 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,983 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:51,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:51,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b3e4b5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:51,988 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:51,988 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,990 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:51,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:51,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:51,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:51,993 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:51,993 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:51,993 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:51,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,996 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:51,996 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:51,996 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:51,996 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:51,996 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:51,996 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:51,996 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:51,998 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:51,998 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,999 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:51,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:51,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:51,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,001 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,007 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,007 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,007 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:52,007 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:52,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:52,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:52,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:52,008 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:52,008 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:52,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:52,008 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:52,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:52,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:52,108 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:52,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:52,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:52,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:52,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:52,115 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:52,115 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:52,115 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:52,115 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:52,115 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,118 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:52,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:52,119 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,119 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:52,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:52,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:52,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:52,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:52,128 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:52,128 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:52,128 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:52,128 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:52,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,131 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,131 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23038ace[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:52,140 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:52,141 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,142 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:52,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:52,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:52,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:52,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:52,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:52,149 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:52,149 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:52,149 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:52,149 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:52,149 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:52,153 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,153 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,153 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:52,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,153 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,157 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,160 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:52,160 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:52,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:52,160 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,160 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:52,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:52,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:52,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:52,161 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:52,162 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:52,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:52,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560492233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:52,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560492255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,262 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:52,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:52,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560492258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560492260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:52,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:52,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560492262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:52,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:52,267 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:52,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:52,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:52,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:52,268 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,268 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,270 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,270 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:52,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:52,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:52,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:52,274 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:52,274 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:52,274 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:52,274 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:52,274 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:52,275 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:52,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:52,278 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,278 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,279 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,279 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43ae64f5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:52,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:52,282 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:52,284 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,284 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:52,285 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,285 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:52,285 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,285 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:52,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:52,290 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:52,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:52,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:52,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:52,290 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:52,293 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,296 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,296 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:52,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:52,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:52,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:52,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:52,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:52,306 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:52,306 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:52,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:52,306 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,306 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:52,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:52,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/d2dc821ec12a44c288e67df49a80f451 2023-04-27 01:53:52,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/2c1ed7dca392474884ce0a0a76aad176 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2c1ed7dca392474884ce0a0a76aad176 2023-04-27 01:53:52,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2c1ed7dca392474884ce0a0a76aad176, entries=150, sequenceid=528, filesize=11.9 K 2023-04-27 01:53:52,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/d146babeb67e4ca2af53225307df6244 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d146babeb67e4ca2af53225307df6244 2023-04-27 01:53:52,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d146babeb67e4ca2af53225307df6244, entries=150, sequenceid=528, filesize=11.9 K 2023-04-27 01:53:52,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/d2dc821ec12a44c288e67df49a80f451 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d2dc821ec12a44c288e67df49a80f451 2023-04-27 01:53:52,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d2dc821ec12a44c288e67df49a80f451, entries=150, sequenceid=528, filesize=11.9 K 2023-04-27 01:53:52,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 4931dc2885b7c715265d908d4b978905 in 1318ms, sequenceid=528, compaction requested=false 2023-04-27 01:53:52,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:52,406 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:52,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:52,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:52,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:52,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:52,411 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:52,411 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:52,411 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:52,411 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:52,411 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,411 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,414 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,414 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:52,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:52,414 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,414 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:52,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:52,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:52,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:52,416 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:52,416 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:52,416 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-04-27 01:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:52,417 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:52,839 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/573d660bc30044d8b6def4ad8d189bfc 2023-04-27 01:53:52,908 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/a30625908115428cb211c283cde737fb 2023-04-27 01:53:52,931 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/cf1626a20c7f42edbe23d8a5888a5a6e 2023-04-27 01:53:52,938 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/573d660bc30044d8b6def4ad8d189bfc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/573d660bc30044d8b6def4ad8d189bfc 2023-04-27 01:53:52,945 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/573d660bc30044d8b6def4ad8d189bfc, entries=150, sequenceid=562, filesize=11.9 K 2023-04-27 01:53:52,946 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/a30625908115428cb211c283cde737fb as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a30625908115428cb211c283cde737fb 2023-04-27 01:53:52,968 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a30625908115428cb211c283cde737fb, entries=150, sequenceid=562, filesize=11.9 K 2023-04-27 01:53:52,969 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/cf1626a20c7f42edbe23d8a5888a5a6e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/cf1626a20c7f42edbe23d8a5888a5a6e 2023-04-27 01:53:52,977 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/cf1626a20c7f42edbe23d8a5888a5a6e, entries=150, sequenceid=562, filesize=11.9 K 2023-04-27 01:53:52,978 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=0 B/0 for 4931dc2885b7c715265d908d4b978905 in 562ms, sequenceid=562, compaction requested=true 2023-04-27 01:53:52,978 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:52,978 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:52,980 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:52,985 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,985 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:52,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dcec2d8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:52,989 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:52,989 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,990 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:52,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:52,991 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:52,991 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:52,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:52,993 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:52,994 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:52,994 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:52,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:52,998 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:52,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:52,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:52,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:52,998 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:53,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,005 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,010 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,014 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,014 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,014 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,015 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:53,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:53,016 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:53,016 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:53,116 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:53,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:53,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:53,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:53,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:53,131 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:53,131 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,131 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:53,131 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:53,139 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,140 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,142 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,142 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:53,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:53,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:53,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:53,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:53,145 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:53,145 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:53,145 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,145 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:53,146 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:53,146 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:53,152 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0f1798[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:53,156 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:53,156 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,158 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,159 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:53,159 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:53,159 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:53,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:53,162 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:53,170 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:53,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:53,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:53,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:53,170 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:53,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,186 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,187 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:53,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,187 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:53,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:53,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:53,287 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:53,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:53,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:53,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:53,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:53,304 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:53,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:53,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:53,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,306 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:53,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:53,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560433295, max timestamp=1682560433308Num uniques 859; 2023-04-27 01:53:53,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560433295, max timestamp=1682560433308Num uniques 859; 2023-04-27 01:53:53,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:53,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:53,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560433295, max timestamp=1682560433308Num uniques 859; 2023-04-27 01:53:53,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:53,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:53,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:53,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:53,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:53,320 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,320 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:53,320 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:53,320 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:53,321 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:53,322 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,322 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,323 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,323 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1565bded[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:53,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:53,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,332 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,332 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:53,333 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:53,333 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:53,335 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,335 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,335 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,337 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560493333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,344 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,344 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,345 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,345 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,345 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,345 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,345 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,345 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:53,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:53,345 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,345 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,346 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:53,346 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:53,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560493339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560493340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560493342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560493444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,446 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:53,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:53,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:53,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:53,452 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:53,452 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,452 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:53,452 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:53,452 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,452 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560493449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560493451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560493451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,455 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:53,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:53,457 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,457 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:53,458 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,458 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:53,459 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:53,459 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:53,462 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25dcfa78[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:53,465 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:53,465 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,466 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:53,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:53,468 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:53,473 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:53,473 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:53,473 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:53,473 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:53,473 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:53,476 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,476 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,476 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,482 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,488 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,488 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,488 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:53,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:53,488 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:53,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:53,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,589 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:53,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:53,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:53,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:53,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:53,595 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:53,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:53,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:53,596 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,596 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,597 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,597 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,601 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,601 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:53,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:53,610 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,610 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:53,610 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:53,610 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:53,612 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,612 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,612 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,612 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e0f2419[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:53,614 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:53,614 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:53,618 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:53,618 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,618 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:53,620 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:53,624 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:53,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:53,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:53,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:53,624 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:53,626 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,626 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,626 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:53,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:53,634 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,635 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:53,635 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:53,635 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,635 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:53,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560493646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560493656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560493658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560493658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/942092d941f940118284729dabb43a6b 2023-04-27 01:53:53,735 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:53,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/8aa99d20895d4bde868eea19599f90e8 2023-04-27 01:53:53,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:53,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:53,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:53,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:53,741 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:53,741 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,741 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:53,741 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:53,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,745 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,746 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,747 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:53,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:53,750 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:53,750 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:53,751 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:53,751 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:53,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/b33b88156f0a4375bd86e659205a0a76 2023-04-27 01:53:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,753 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:53,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cee4597[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:53,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:53,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/942092d941f940118284729dabb43a6b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/942092d941f940118284729dabb43a6b 2023-04-27 01:53:53,761 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:53,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:53,762 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,762 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:53,764 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,765 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/942092d941f940118284729dabb43a6b, entries=200, sequenceid=573, filesize=14.3 K 2023-04-27 01:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/8aa99d20895d4bde868eea19599f90e8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/8aa99d20895d4bde868eea19599f90e8 2023-04-27 01:53:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:53,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:53,768 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:53,768 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:53,768 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:53,769 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:53,769 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:53,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/8aa99d20895d4bde868eea19599f90e8, entries=100, sequenceid=573, filesize=9.5 K 2023-04-27 01:53:53,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/b33b88156f0a4375bd86e659205a0a76 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/b33b88156f0a4375bd86e659205a0a76 2023-04-27 01:53:53,778 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,778 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,779 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:53,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:53,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,786 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:53,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:53,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:53,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:53,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:53,788 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:53,788 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:53,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/b33b88156f0a4375bd86e659205a0a76, entries=100, sequenceid=573, filesize=9.5 K 2023-04-27 01:53:53,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4931dc2885b7c715265d908d4b978905 in 527ms, sequenceid=573, compaction requested=true 2023-04-27 01:53:53,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:53,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:53,791 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:53,791 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:53,794 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51556 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:53,794 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46676 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:53,795 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:53,795 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,795 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/175b277e76a642dc9d37d5f864339d12, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d146babeb67e4ca2af53225307df6244, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a30625908115428cb211c283cde737fb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/8aa99d20895d4bde868eea19599f90e8] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=45.6 K 2023-04-27 01:53:53,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:53,794 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:53,797 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,797 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85199c1ad627440ba207ed0d44123d65, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2c1ed7dca392474884ce0a0a76aad176, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/573d660bc30044d8b6def4ad8d189bfc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/942092d941f940118284729dabb43a6b] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=50.3 K 2023-04-27 01:53:53,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:53,798 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 85199c1ad627440ba207ed0d44123d65, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682560430114 2023-04-27 01:53:53,799 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 175b277e76a642dc9d37d5f864339d12, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682560430114 2023-04-27 01:53:53,800 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2c1ed7dca392474884ce0a0a76aad176, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682560430737 2023-04-27 01:53:53,800 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d146babeb67e4ca2af53225307df6244, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682560430737 2023-04-27 01:53:53,801 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting a30625908115428cb211c283cde737fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1682560431130 2023-04-27 01:53:53,801 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 573d660bc30044d8b6def4ad8d189bfc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1682560431130 2023-04-27 01:53:53,801 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8aa99d20895d4bde868eea19599f90e8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682560433259 2023-04-27 01:53:53,802 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 942092d941f940118284729dabb43a6b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682560433253 2023-04-27 01:53:53,817 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:53,826 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:53,856 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/2e97386bc79b4dfbad4cc2651a92933e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2e97386bc79b4dfbad4cc2651a92933e 2023-04-27 01:53:53,866 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 2e97386bc79b4dfbad4cc2651a92933e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:53,866 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:53,866 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=12, startTime=1682560433791; duration=0sec 2023-04-27 01:53:53,867 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:53,867 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:53:53,868 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46676 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:53:53,868 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:53,868 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,869 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c24b5c3f428b49559a1b8460412cd2e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d2dc821ec12a44c288e67df49a80f451, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/cf1626a20c7f42edbe23d8a5888a5a6e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/b33b88156f0a4375bd86e659205a0a76] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=45.6 K 2023-04-27 01:53:53,869 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c24b5c3f428b49559a1b8460412cd2e5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682560430114 2023-04-27 01:53:53,871 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d2dc821ec12a44c288e67df49a80f451, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682560430737 2023-04-27 01:53:53,872 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cf1626a20c7f42edbe23d8a5888a5a6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1682560431130 2023-04-27 01:53:53,873 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b33b88156f0a4375bd86e659205a0a76, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682560433259 2023-04-27 01:53:53,888 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:53,892 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#47 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:53,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:53,907 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:53,907 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,907 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:53,907 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:53,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,909 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:53,909 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/80bfb4fb56584b62b2bec7e3870a1fd2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/80bfb4fb56584b62b2bec7e3870a1fd2 2023-04-27 01:53:53,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:53,909 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:53,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:53,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:53,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:53,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:53,916 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 01:53:53,917 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 80bfb4fb56584b62b2bec7e3870a1fd2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:53,917 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:53,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:53,917 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=12, startTime=1682560433796; duration=0sec 2023-04-27 01:53:53,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:53,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:53,917 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:53,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:53,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:53,918 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:53,946 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/ee1de37a00144ee086f5cbf2040a0aae 2023-04-27 01:53:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:53,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:53,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560493969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560493972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560493972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:53,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:53,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560493975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560494078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560494079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560494079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,251 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/1efdb21e9ed1494f9609b317417e3497 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/1efdb21e9ed1494f9609b317417e3497 2023-04-27 01:53:54,268 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into 1efdb21e9ed1494f9609b317417e3497(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:54,268 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:54,268 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=12, startTime=1682560433791; duration=0sec 2023-04-27 01:53:54,269 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:54,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560494278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,285 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:53:54,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560494288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560494288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560494289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,367 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/991792dfcfb34893a0b02fccc409ff69 2023-04-27 01:53:54,396 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/4fbdcd5869d14c539e82454b304f7312 2023-04-27 01:53:54,408 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/ee1de37a00144ee086f5cbf2040a0aae as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/ee1de37a00144ee086f5cbf2040a0aae 2023-04-27 01:53:54,423 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/ee1de37a00144ee086f5cbf2040a0aae, entries=150, sequenceid=611, filesize=11.9 K 2023-04-27 01:53:54,425 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/991792dfcfb34893a0b02fccc409ff69 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/991792dfcfb34893a0b02fccc409ff69 2023-04-27 01:53:54,431 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/991792dfcfb34893a0b02fccc409ff69, entries=150, sequenceid=611, filesize=11.9 K 2023-04-27 01:53:54,432 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/4fbdcd5869d14c539e82454b304f7312 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/4fbdcd5869d14c539e82454b304f7312 2023-04-27 01:53:54,438 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/4fbdcd5869d14c539e82454b304f7312, entries=150, sequenceid=611, filesize=11.9 K 2023-04-27 01:53:54,439 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 4931dc2885b7c715265d908d4b978905 in 523ms, sequenceid=611, compaction requested=false 2023-04-27 01:53:54,439 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:54,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:54,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:53:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:54,513 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445c1532[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:54,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:54,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560434516, max timestamp=1682560434527Num uniques 859; 2023-04-27 01:53:54,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:54,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560434516, max timestamp=1682560434527Num uniques 859; 2023-04-27 01:53:54,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:54,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560434516, max timestamp=1682560434527Num uniques 859; 2023-04-27 01:53:54,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:53:54,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:54,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:54,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:54,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:54,590 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:54,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560494590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:54,593 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:54,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:54,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:54,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:54,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,594 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560494591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560494591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560494596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,614 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,636 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,636 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:54,636 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:54,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:54,636 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:54,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:54,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:54,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:54,636 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,637 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,637 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,637 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,637 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,637 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:54,637 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:54,637 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:54,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560494695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,737 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:54,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:54,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:54,742 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:54,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:54,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:54,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:54,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,743 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:54,743 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:54,744 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,744 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:54,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:54,746 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:54,746 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:54,746 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:54,746 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:54,748 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69128fb[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:54,751 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:54,752 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,753 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:54,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:54,753 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,753 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:54,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:54,760 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:54,780 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:54,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:54,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:54,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:54,780 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:54,782 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,782 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:54,782 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,786 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:54,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:54,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:54,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:54,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:54,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:54,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:54,790 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:54,790 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:54,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:54,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:54,890 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:54,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:54,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:54,894 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:54,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:54,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:54,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:54,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,896 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:54,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:54,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/4dc48826045945d792071b532a6ed473 2023-04-27 01:53:54,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:54,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:54,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:54,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:54,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:54,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:54,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:54,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:54,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:54,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560494900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:54,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51e9fbdc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:54,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:54,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,909 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:54,911 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,911 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:54,911 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:54,912 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:54,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:54,915 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:54,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:54,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:54,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:54,915 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:54,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:54,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:54,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:54,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,920 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,925 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:54,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:54,925 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,925 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:54,925 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:54,925 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:54,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/d47002afc7f641a78407c95e23eea2ae 2023-04-27 01:53:54,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:54,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:54,929 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:54,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:54,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:54,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:54,929 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:54,929 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:54,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:54,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/0649f098b0234849bd336d6142e06f6a 2023-04-27 01:53:54,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/4dc48826045945d792071b532a6ed473 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4dc48826045945d792071b532a6ed473 2023-04-27 01:53:54,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4dc48826045945d792071b532a6ed473, entries=150, sequenceid=623, filesize=11.9 K 2023-04-27 01:53:55,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/d47002afc7f641a78407c95e23eea2ae as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d47002afc7f641a78407c95e23eea2ae 2023-04-27 01:53:55,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d47002afc7f641a78407c95e23eea2ae, entries=100, sequenceid=623, filesize=9.5 K 2023-04-27 01:53:55,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/0649f098b0234849bd336d6142e06f6a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0649f098b0234849bd336d6142e06f6a 2023-04-27 01:53:55,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0649f098b0234849bd336d6142e06f6a, entries=100, sequenceid=623, filesize=9.5 K 2023-04-27 01:53:55,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4931dc2885b7c715265d908d4b978905 in 535ms, sequenceid=623, compaction requested=true 2023-04-27 01:53:55,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:55,016 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:55,016 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:55,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:55,018 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37066 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:55,018 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:55,018 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:55,018 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:55,018 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:55,018 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:55,018 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2e97386bc79b4dfbad4cc2651a92933e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/ee1de37a00144ee086f5cbf2040a0aae, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4dc48826045945d792071b532a6ed473] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=36.2 K 2023-04-27 01:53:55,019 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/1efdb21e9ed1494f9609b317417e3497, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/991792dfcfb34893a0b02fccc409ff69, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d47002afc7f641a78407c95e23eea2ae] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=33.8 K 2023-04-27 01:53:55,019 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2e97386bc79b4dfbad4cc2651a92933e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682560431138 2023-04-27 01:53:55,020 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1efdb21e9ed1494f9609b317417e3497, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682560431138 2023-04-27 01:53:55,022 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting ee1de37a00144ee086f5cbf2040a0aae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1682560433331 2023-04-27 01:53:55,022 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 991792dfcfb34893a0b02fccc409ff69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1682560433331 2023-04-27 01:53:55,022 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 4dc48826045945d792071b532a6ed473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682560433971 2023-04-27 01:53:55,023 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d47002afc7f641a78407c95e23eea2ae, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682560433974 2023-04-27 01:53:55,030 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:55,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:55,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:55,041 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:55,044 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:55,044 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:55,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:55,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:55,046 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:55,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:55,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:55,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:55,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:55,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:55,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:55,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:55,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:55,067 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:55,067 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:55,067 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:55,068 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-27 01:53:55,068 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:55,068 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:55,068 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:55,068 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:55,068 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:55,068 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:55,090 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/56b7f525217e412892326375287e6942 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/56b7f525217e412892326375287e6942 2023-04-27 01:53:55,100 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/9a9dd9eb814c4ef7b7c9b2b9f7f67cdf as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/9a9dd9eb814c4ef7b7c9b2b9f7f67cdf 2023-04-27 01:53:55,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:55,103 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:55,103 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 56b7f525217e412892326375287e6942(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:55,103 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:55,104 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=13, startTime=1682560435016; duration=0sec 2023-04-27 01:53:55,104 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:55,104 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:55,107 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34626 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:55,108 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:55,108 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:55,108 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/80bfb4fb56584b62b2bec7e3870a1fd2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/4fbdcd5869d14c539e82454b304f7312, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0649f098b0234849bd336d6142e06f6a] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=33.8 K 2023-04-27 01:53:55,108 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 80bfb4fb56584b62b2bec7e3870a1fd2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682560431138 2023-04-27 01:53:55,109 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 4fbdcd5869d14c539e82454b304f7312, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1682560433331 2023-04-27 01:53:55,109 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 0649f098b0234849bd336d6142e06f6a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682560433974 2023-04-27 01:53:55,121 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into 9a9dd9eb814c4ef7b7c9b2b9f7f67cdf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:55,123 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:55,123 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=13, startTime=1682560435016; duration=0sec 2023-04-27 01:53:55,123 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:55,129 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:55,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560495128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560495128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560495128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560495221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560495234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560495235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560495235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560495439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560495439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560495440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,520 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/30ad44c6dd034a22a09e52088bf48d9b 2023-04-27 01:53:55,578 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/6ce1dab730f546848203949c3b73da86 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6ce1dab730f546848203949c3b73da86 2023-04-27 01:53:55,580 WARN [DataStreamer for file /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/b525a104a8a84ed4a37d15a59ffc2c7d] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-27 01:53:55,581 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/b525a104a8a84ed4a37d15a59ffc2c7d 2023-04-27 01:53:55,589 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 6ce1dab730f546848203949c3b73da86(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:55,589 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:55,589 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=13, startTime=1682560435016; duration=0sec 2023-04-27 01:53:55,589 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:55,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560495725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560495743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560495744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:55,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:55,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560495745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,023 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/21d7ee2094d74e71a9be6ddd6b000f9e 2023-04-27 01:53:56,031 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/30ad44c6dd034a22a09e52088bf48d9b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/30ad44c6dd034a22a09e52088bf48d9b 2023-04-27 01:53:56,041 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/30ad44c6dd034a22a09e52088bf48d9b, entries=150, sequenceid=658, filesize=11.9 K 2023-04-27 01:53:56,044 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/b525a104a8a84ed4a37d15a59ffc2c7d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b525a104a8a84ed4a37d15a59ffc2c7d 2023-04-27 01:53:56,054 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b525a104a8a84ed4a37d15a59ffc2c7d, entries=150, sequenceid=658, filesize=11.9 K 2023-04-27 01:53:56,055 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/21d7ee2094d74e71a9be6ddd6b000f9e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21d7ee2094d74e71a9be6ddd6b000f9e 2023-04-27 01:53:56,066 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21d7ee2094d74e71a9be6ddd6b000f9e, entries=150, sequenceid=658, filesize=11.9 K 2023-04-27 01:53:56,067 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 4931dc2885b7c715265d908d4b978905 in 999ms, sequenceid=658, compaction requested=false 2023-04-27 01:53:56,067 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:56,067 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:56,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:56,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:56,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:56,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f25de8b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:56,075 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:56,075 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:56,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:56,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:56,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:56,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:56,083 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:56,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:56,087 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:56,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:56,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:56,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:56,087 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:56,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:56,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:56,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,090 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:56,092 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:56,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:56,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:56,096 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:56,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:56,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:56,096 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:56,096 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:56,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:56,196 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:56,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:56,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:56,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:56,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:56,204 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:56,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:56,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:56,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:56,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,205 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:56,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:56,206 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:56,206 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,206 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:56,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:56,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:56,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:56,209 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:56,209 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:56,209 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:53:56,209 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:56,209 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:56,210 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:56,210 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:56,210 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:56,210 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:56,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:56,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:56,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560496272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560496275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682560436261, max timestamp=1682560436275Num uniques 859; 2023-04-27 01:53:56,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-27 01:53:56,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560436268, max timestamp=1682560436275Num uniques 859; 2023-04-27 01:53:56,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:53:56,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682560436261, max timestamp=1682560436275Num uniques 859; 2023-04-27 01:53:56,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-27 01:53:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560496315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560496378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560496379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560496418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560496582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560496589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560496621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,634 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/3932051de581419bb76203cb2f6b08ab 2023-04-27 01:53:56,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560496737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560496886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560496891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:56,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:56,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560496929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,065 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/86519a386c434acc8c8aecb1d7b34824 2023-04-27 01:53:57,082 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/1ea2b1199fb247f094b5333de598ce77 2023-04-27 01:53:57,088 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/3932051de581419bb76203cb2f6b08ab as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3932051de581419bb76203cb2f6b08ab 2023-04-27 01:53:57,093 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3932051de581419bb76203cb2f6b08ab, entries=150, sequenceid=672, filesize=11.9 K 2023-04-27 01:53:57,094 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/86519a386c434acc8c8aecb1d7b34824 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/86519a386c434acc8c8aecb1d7b34824 2023-04-27 01:53:57,099 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/86519a386c434acc8c8aecb1d7b34824, entries=150, sequenceid=672, filesize=11.9 K 2023-04-27 01:53:57,100 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/1ea2b1199fb247f094b5333de598ce77 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1ea2b1199fb247f094b5333de598ce77 2023-04-27 01:53:57,105 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1ea2b1199fb247f094b5333de598ce77, entries=150, sequenceid=672, filesize=11.9 K 2023-04-27 01:53:57,106 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for 4931dc2885b7c715265d908d4b978905 in 897ms, sequenceid=672, compaction requested=true 2023-04-27 01:53:57,106 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:57,106 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:57,109 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:57,109 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,109 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:57,109 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:57,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,110 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:57,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b74b5d8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:57,113 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:57,113 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,114 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:57,114 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,114 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:57,117 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,117 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,117 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,117 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:57,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:57,120 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:57,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:57,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:57,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:57,120 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:57,122 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,122 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,122 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,125 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:57,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:57,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:57,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:57,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:57,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:57,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:57,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,131 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:57,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:57,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:57,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:57,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:57,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:57,235 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:57,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:57,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:57,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:57,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,236 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,236 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:57,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:57,237 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,237 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:57,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:57,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:57,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:57,239 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=131.94 KB heapSize=365.84 KB 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:57,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:53:57,249 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/12c18a93e6724fbdbbfcaa78656759b6 2023-04-27 01:53:57,265 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/d63890185ada486fbe22244d1f722d6e 2023-04-27 01:53:57,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:57,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:57,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560497407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560497408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560497437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560497512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560497513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,678 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=706 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/c03e6d134dcb4f84a9051ca8718d8dc3 2023-04-27 01:53:57,684 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/12c18a93e6724fbdbbfcaa78656759b6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/12c18a93e6724fbdbbfcaa78656759b6 2023-04-27 01:53:57,689 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/12c18a93e6724fbdbbfcaa78656759b6, entries=150, sequenceid=706, filesize=11.9 K 2023-04-27 01:53:57,690 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/d63890185ada486fbe22244d1f722d6e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d63890185ada486fbe22244d1f722d6e 2023-04-27 01:53:57,695 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d63890185ada486fbe22244d1f722d6e, entries=150, sequenceid=706, filesize=11.9 K 2023-04-27 01:53:57,696 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/c03e6d134dcb4f84a9051ca8718d8dc3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c03e6d134dcb4f84a9051ca8718d8dc3 2023-04-27 01:53:57,701 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c03e6d134dcb4f84a9051ca8718d8dc3, entries=150, sequenceid=706, filesize=11.9 K 2023-04-27 01:53:57,702 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~365.80 KB/374576, currentSize=53.67 KB/54960 for 4931dc2885b7c715265d908d4b978905 in 463ms, sequenceid=706, compaction requested=true 2023-04-27 01:53:57,702 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:57,702 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:57,704 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:57,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ad9cbd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:57,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:57,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,718 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:57,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:57,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:57,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:53:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:57,720 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:57,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:57,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:57,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:57,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:57,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:57,730 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:57,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:57,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:57,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:57,730 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:57,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:57,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,733 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,736 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,739 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,739 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,739 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:57,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:57,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:57,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:57,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:57,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:57,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:57,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:57,742 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:57,742 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:57,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:57,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560497756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560497757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,842 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:57,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:57,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:57,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:57,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:57,848 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:57,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:57,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:57,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:57,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,853 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:57,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,854 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:57,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:57,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:57,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:57,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560497859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:57,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560497861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:57,864 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:57,864 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:57,864 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:57,864 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a04beef[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:57,868 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:57,868 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,869 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:57,870 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,870 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:57,872 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,872 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,872 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:57,872 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:57,878 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:57,878 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:57,878 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:57,878 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:57,879 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:57,880 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,880 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:57,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,880 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:57,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,882 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,888 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:57,888 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:57,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:57,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:57,896 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:57,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:57,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:57,897 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:57,897 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:57,896 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:57,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:57,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:57,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:57,997 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:57,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:58,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:58,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:58,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:58,002 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:58,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:58,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:58,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,003 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:58,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:58,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:58,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:58,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:58,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:58,005 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,006 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:58,006 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:58,006 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:58,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:58,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:58,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:58,007 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b24f4a2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:58,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:58,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,013 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:58,013 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:58,013 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:58,013 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,013 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,014 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:58,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:58,019 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:58,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:58,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:58,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:58,019 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:58,021 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,021 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,022 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,023 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,024 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,031 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:58,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:58,031 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,032 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:58,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560498064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560498064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,132 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:58,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:58,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:58,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:58,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:58,135 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:58,136 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,136 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:58,136 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:58,136 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,136 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,137 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,137 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:58,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:58,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:58,139 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,139 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:58,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:58,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:58,143 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,143 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,143 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,143 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4520586b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:58,145 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:58,145 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:58,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:58,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/4ee47c9e1b8c4e6a82aabbeef104834e 2023-04-27 01:53:58,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:58,147 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,148 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,148 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:58,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:58,155 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:58,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:58,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:58,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:58,155 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:58,157 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,157 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,157 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,158 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,160 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/356976fd419a46e4bb9c34f99de47fc5 2023-04-27 01:53:58,163 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:58,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,164 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,164 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,164 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:58,164 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:58,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:58,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/d66921f4ad724c6cb9379fc07cc76967 2023-04-27 01:53:58,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/4ee47c9e1b8c4e6a82aabbeef104834e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4ee47c9e1b8c4e6a82aabbeef104834e 2023-04-27 01:53:58,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4ee47c9e1b8c4e6a82aabbeef104834e, entries=250, sequenceid=719, filesize=16.7 K 2023-04-27 01:53:58,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/356976fd419a46e4bb9c34f99de47fc5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/356976fd419a46e4bb9c34f99de47fc5 2023-04-27 01:53:58,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/356976fd419a46e4bb9c34f99de47fc5, entries=150, sequenceid=719, filesize=11.9 K 2023-04-27 01:53:58,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/d66921f4ad724c6cb9379fc07cc76967 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d66921f4ad724c6cb9379fc07cc76967 2023-04-27 01:53:58,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d66921f4ad724c6cb9379fc07cc76967, entries=150, sequenceid=719, filesize=11.9 K 2023-04-27 01:53:58,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4931dc2885b7c715265d908d4b978905 in 501ms, sequenceid=719, compaction requested=true 2023-04-27 01:53:58,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:58,220 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:58,220 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:53:58,220 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:53:58,220 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:58,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:58,226 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66424 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:53:58,227 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:53:58,227 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:58,227 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,227 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/9a9dd9eb814c4ef7b7c9b2b9f7f67cdf, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b525a104a8a84ed4a37d15a59ffc2c7d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/86519a386c434acc8c8aecb1d7b34824, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d63890185ada486fbe22244d1f722d6e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/356976fd419a46e4bb9c34f99de47fc5] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=60.1 K 2023-04-27 01:53:58,227 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:58,227 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,227 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/56b7f525217e412892326375287e6942, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/30ad44c6dd034a22a09e52088bf48d9b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3932051de581419bb76203cb2f6b08ab, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/12c18a93e6724fbdbbfcaa78656759b6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4ee47c9e1b8c4e6a82aabbeef104834e] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=64.9 K 2023-04-27 01:53:58,228 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9a9dd9eb814c4ef7b7c9b2b9f7f67cdf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682560433331 2023-04-27 01:53:58,228 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 56b7f525217e412892326375287e6942, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682560433331 2023-04-27 01:53:58,228 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b525a104a8a84ed4a37d15a59ffc2c7d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682560434579 2023-04-27 01:53:58,229 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 30ad44c6dd034a22a09e52088bf48d9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682560434579 2023-04-27 01:53:58,229 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3932051de581419bb76203cb2f6b08ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682560435101 2023-04-27 01:53:58,229 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 86519a386c434acc8c8aecb1d7b34824, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682560435101 2023-04-27 01:53:58,229 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 12c18a93e6724fbdbbfcaa78656759b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1682560436298 2023-04-27 01:53:58,230 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d63890185ada486fbe22244d1f722d6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1682560436298 2023-04-27 01:53:58,230 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 4ee47c9e1b8c4e6a82aabbeef104834e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682560437404 2023-04-27 01:53:58,230 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 356976fd419a46e4bb9c34f99de47fc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682560437406 2023-04-27 01:53:58,250 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#69 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:58,252 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:58,265 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:58,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:58,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:58,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:58,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:58,287 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:58,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:58,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:58,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,294 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,294 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:58,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:58,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:58,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:58,299 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:53:58,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:58,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:58,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:58,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,304 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/0c635d15740b4f51a1b40fde59a505bd as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/0c635d15740b4f51a1b40fde59a505bd 2023-04-27 01:53:58,312 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into 0c635d15740b4f51a1b40fde59a505bd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:58,313 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:58,313 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=11, startTime=1682560438220; duration=0sec 2023-04-27 01:53:58,313 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:58,313 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:53:58,315 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:53:58,315 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:58,315 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,315 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6ce1dab730f546848203949c3b73da86, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21d7ee2094d74e71a9be6ddd6b000f9e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1ea2b1199fb247f094b5333de598ce77, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c03e6d134dcb4f84a9051ca8718d8dc3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d66921f4ad724c6cb9379fc07cc76967] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=60.1 K 2023-04-27 01:53:58,316 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6ce1dab730f546848203949c3b73da86, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1682560433331 2023-04-27 01:53:58,316 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 21d7ee2094d74e71a9be6ddd6b000f9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682560434579 2023-04-27 01:53:58,317 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1ea2b1199fb247f094b5333de598ce77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682560435101 2023-04-27 01:53:58,317 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c03e6d134dcb4f84a9051ca8718d8dc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=706, earliestPutTs=1682560436298 2023-04-27 01:53:58,318 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d66921f4ad724c6cb9379fc07cc76967, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682560437406 2023-04-27 01:53:58,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:58,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:58,329 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/88284bf7db164b13bceff1cd623a2e9a 2023-04-27 01:53:58,330 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#72 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:58,358 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/abd9768649e94478b12d5c3b20686ec7 2023-04-27 01:53:58,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560498355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560498367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560498369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,394 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/3d9e78a97c6f41a9ad864c570cee2769 2023-04-27 01:53:58,401 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/88284bf7db164b13bceff1cd623a2e9a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/88284bf7db164b13bceff1cd623a2e9a 2023-04-27 01:53:58,406 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/88284bf7db164b13bceff1cd623a2e9a, entries=150, sequenceid=738, filesize=11.9 K 2023-04-27 01:53:58,407 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/abd9768649e94478b12d5c3b20686ec7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/abd9768649e94478b12d5c3b20686ec7 2023-04-27 01:53:58,412 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/abd9768649e94478b12d5c3b20686ec7, entries=150, sequenceid=738, filesize=11.9 K 2023-04-27 01:53:58,413 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/3d9e78a97c6f41a9ad864c570cee2769 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d9e78a97c6f41a9ad864c570cee2769 2023-04-27 01:53:58,417 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d9e78a97c6f41a9ad864c570cee2769, entries=150, sequenceid=738, filesize=11.9 K 2023-04-27 01:53:58,418 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4931dc2885b7c715265d908d4b978905 in 119ms, sequenceid=738, compaction requested=false 2023-04-27 01:53:58,418 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:58,419 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:58,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,423 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d707c06[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:58,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:58,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,428 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:58,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:58,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:58,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:58,434 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:58,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:58,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:58,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:58,434 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:58,435 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,435 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,435 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:58,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,440 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:58,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,441 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:58,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:58,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:58,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:53:58,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:58,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:58,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:58,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560498497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560498498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,541 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:58,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:58,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:58,545 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:58,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:58,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:58,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,547 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,547 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:58,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:58,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:58,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:58,551 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,551 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:58,551 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:58,552 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:58,554 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@124c5281[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:58,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:58,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,558 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:58,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:58,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:58,562 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:58,562 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:58,562 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:58,562 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:58,562 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:58,563 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,563 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,566 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,569 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,569 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,569 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:58,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:58,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,570 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:58,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:58,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,570 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:58,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560498601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560498604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,670 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:58,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:58,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:58,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:58,675 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:58,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:58,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:58,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:58,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:58,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:58,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:58,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:58,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:58,690 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/2ba569db7e0b45b6866b8b5ad64439cc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2ba569db7e0b45b6866b8b5ad64439cc 2023-04-27 01:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef1358e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:58,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:58,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:58,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:58,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:58,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:58,698 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,700 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 2ba569db7e0b45b6866b8b5ad64439cc(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:58,700 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:58,700 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=11, startTime=1682560438220; duration=0sec 2023-04-27 01:53:58,700 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:58,704 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:58,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:58,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:58,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:58,704 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:58,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,712 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,713 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:58,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:58,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:58,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:58,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,718 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,718 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,718 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,718 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:58,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560498743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,747 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:53:58,758 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/934e3e451c874ceeaa13668b4325137f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/934e3e451c874ceeaa13668b4325137f 2023-04-27 01:53:58,766 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 934e3e451c874ceeaa13668b4325137f(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:58,766 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:58,766 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=11, startTime=1682560438221; duration=0sec 2023-04-27 01:53:58,766 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:58,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560498809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560498810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:58,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:58,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:58,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:58,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:58,824 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:58,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:58,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:58,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:58,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:58,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:58,832 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,832 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:58,833 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:58,833 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:58,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,835 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,835 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,835 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69d5d33b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:58,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:58,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:58,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:58,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:58,843 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:58,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:58,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,845 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:58,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:58,850 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:58,850 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:58,850 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:58,850 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:58,850 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:58,853 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,853 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,853 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,858 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:58,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/3799d4e39ad640fd99898709502d1f8a 2023-04-27 01:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,872 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,872 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:53:58,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:53:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:53:58,874 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,874 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,874 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,874 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,874 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,874 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:58,874 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:58,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:53:58,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:53:58,874 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:53:58,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:53:58,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:53:58,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560498873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:58,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560498875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/866cb85b326e4200a5af7817c0045ee9 2023-04-27 01:53:58,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/11020a3daf3c4b3dbbe1f0b5984bbdc6 2023-04-27 01:53:58,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/3799d4e39ad640fd99898709502d1f8a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3799d4e39ad640fd99898709502d1f8a 2023-04-27 01:53:58,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3799d4e39ad640fd99898709502d1f8a, entries=150, sequenceid=753, filesize=11.9 K 2023-04-27 01:53:58,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/866cb85b326e4200a5af7817c0045ee9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/866cb85b326e4200a5af7817c0045ee9 2023-04-27 01:53:58,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/866cb85b326e4200a5af7817c0045ee9, entries=150, sequenceid=753, filesize=11.9 K 2023-04-27 01:53:58,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/11020a3daf3c4b3dbbe1f0b5984bbdc6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/11020a3daf3c4b3dbbe1f0b5984bbdc6 2023-04-27 01:53:58,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/11020a3daf3c4b3dbbe1f0b5984bbdc6, entries=150, sequenceid=753, filesize=11.9 K 2023-04-27 01:53:58,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 4931dc2885b7c715265d908d4b978905 in 507ms, sequenceid=753, compaction requested=true 2023-04-27 01:53:58,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:58,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:58,958 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:58,958 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:58,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:58,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:58,959 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:58,960 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:58,960 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:58,960 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:58,960 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,960 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,960 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2ba569db7e0b45b6866b8b5ad64439cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/88284bf7db164b13bceff1cd623a2e9a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3799d4e39ad640fd99898709502d1f8a] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=36.5 K 2023-04-27 01:53:58,960 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/0c635d15740b4f51a1b40fde59a505bd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/abd9768649e94478b12d5c3b20686ec7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/866cb85b326e4200a5af7817c0045ee9] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=36.5 K 2023-04-27 01:53:58,960 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2ba569db7e0b45b6866b8b5ad64439cc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682560437406 2023-04-27 01:53:58,960 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 0c635d15740b4f51a1b40fde59a505bd, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682560437406 2023-04-27 01:53:58,961 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 88284bf7db164b13bceff1cd623a2e9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1682560437753 2023-04-27 01:53:58,961 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting abd9768649e94478b12d5c3b20686ec7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1682560437753 2023-04-27 01:53:58,961 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3799d4e39ad640fd99898709502d1f8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682560438333 2023-04-27 01:53:58,961 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 866cb85b326e4200a5af7817c0045ee9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682560438333 2023-04-27 01:53:58,969 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:58,973 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:58,975 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:53:58,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:53:58,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:53:58,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:53:58,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:53:58,991 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:53:58,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:53:58,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:53:58,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,994 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,994 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:53:58,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:53:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:53:58,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:58,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:53:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:53:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:53:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:53:58,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:53:58,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:53:58,996 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:58,996 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:53:58,996 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:53:58,997 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:58,997 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,997 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:58,997 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:58,997 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:58,997 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:59,004 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/77bd45e707cf45f8b88e8680b51c39a5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/77bd45e707cf45f8b88e8680b51c39a5 2023-04-27 01:53:59,018 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 77bd45e707cf45f8b88e8680b51c39a5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:59,018 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:59,018 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=13, startTime=1682560438958; duration=0sec 2023-04-27 01:53:59,019 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:59,019 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:59,020 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:59,021 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:59,021 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:59,021 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/934e3e451c874ceeaa13668b4325137f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d9e78a97c6f41a9ad864c570cee2769, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/11020a3daf3c4b3dbbe1f0b5984bbdc6] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=36.5 K 2023-04-27 01:53:59,021 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 934e3e451c874ceeaa13668b4325137f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1682560437406 2023-04-27 01:53:59,022 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3d9e78a97c6f41a9ad864c570cee2769, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=738, earliestPutTs=1682560437753 2023-04-27 01:53:59,023 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 11020a3daf3c4b3dbbe1f0b5984bbdc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682560438333 2023-04-27 01:53:59,033 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:59,056 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/7104096c4aa142d4afb7c33abd1c594f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7104096c4aa142d4afb7c33abd1c594f 2023-04-27 01:53:59,080 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 7104096c4aa142d4afb7c33abd1c594f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:59,080 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:59,080 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=13, startTime=1682560438958; duration=0sec 2023-04-27 01:53:59,080 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:59,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:59,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:53:59,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560499126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560499128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560499229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560499230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,401 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/b7f03d2bf9b54143bc500bb34579bac1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b7f03d2bf9b54143bc500bb34579bac1 2023-04-27 01:53:59,407 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into b7f03d2bf9b54143bc500bb34579bac1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:59,407 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:59,407 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=13, startTime=1682560438958; duration=0sec 2023-04-27 01:53:59,408 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:59,409 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/9a2f713ed879442ebda4224af1ef3e03 2023-04-27 01:53:59,421 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/55b5812310a24706bcc5d491dcf0c6b3 2023-04-27 01:53:59,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560499431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560499433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560499734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560499738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,845 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/21f58f0aa53e453ab6fa7e75a50a733f 2023-04-27 01:53:59,853 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/9a2f713ed879442ebda4224af1ef3e03 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9a2f713ed879442ebda4224af1ef3e03 2023-04-27 01:53:59,858 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9a2f713ed879442ebda4224af1ef3e03, entries=150, sequenceid=774, filesize=11.9 K 2023-04-27 01:53:59,859 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/55b5812310a24706bcc5d491dcf0c6b3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/55b5812310a24706bcc5d491dcf0c6b3 2023-04-27 01:53:59,867 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/55b5812310a24706bcc5d491dcf0c6b3, entries=150, sequenceid=774, filesize=11.9 K 2023-04-27 01:53:59,868 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/21f58f0aa53e453ab6fa7e75a50a733f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21f58f0aa53e453ab6fa7e75a50a733f 2023-04-27 01:53:59,873 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21f58f0aa53e453ab6fa7e75a50a733f, entries=150, sequenceid=774, filesize=11.9 K 2023-04-27 01:53:59,874 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4931dc2885b7c715265d908d4b978905 in 878ms, sequenceid=774, compaction requested=false 2023-04-27 01:53:59,874 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:59,874 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:59,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:53:59,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:53:59,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:53:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:53:59,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:53:59,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:53:59,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:59,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:53:59,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:59,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:53:59,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:53:59,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/254aa9de247540ceb0c04426d0a1f856 2023-04-27 01:53:59,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560499901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:53:59,905 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:59,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560499904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:59,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:53:59,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371b124b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:53:59,907 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:53:59,907 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:59,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/351f83cb9f8e4d92a54ac3e4af22cafc 2023-04-27 01:53:59,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=791 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/eeba3c1bb4e846d182a29508f729c2f4 2023-04-27 01:53:59,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/254aa9de247540ceb0c04426d0a1f856 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/254aa9de247540ceb0c04426d0a1f856 2023-04-27 01:53:59,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/254aa9de247540ceb0c04426d0a1f856, entries=200, sequenceid=791, filesize=14.3 K 2023-04-27 01:53:59,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/351f83cb9f8e4d92a54ac3e4af22cafc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351f83cb9f8e4d92a54ac3e4af22cafc 2023-04-27 01:53:59,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:59,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:59,943 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,944 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:53:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:53:59,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:53:59,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:53:59,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:53:59,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351f83cb9f8e4d92a54ac3e4af22cafc, entries=150, sequenceid=791, filesize=11.9 K 2023-04-27 01:53:59,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/eeba3c1bb4e846d182a29508f729c2f4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/eeba3c1bb4e846d182a29508f729c2f4 2023-04-27 01:53:59,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/eeba3c1bb4e846d182a29508f729c2f4, entries=150, sequenceid=791, filesize=11.9 K 2023-04-27 01:53:59,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4931dc2885b7c715265d908d4b978905 in 72ms, sequenceid=791, compaction requested=true 2023-04-27 01:53:59,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:53:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:59,955 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:59,955 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:59,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:59,956 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:59,956 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:53:59,956 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:59,956 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:59,956 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:53:59,956 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/77bd45e707cf45f8b88e8680b51c39a5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9a2f713ed879442ebda4224af1ef3e03, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/254aa9de247540ceb0c04426d0a1f856] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=38.9 K 2023-04-27 01:53:59,956 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:59,956 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b7f03d2bf9b54143bc500bb34579bac1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/55b5812310a24706bcc5d491dcf0c6b3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351f83cb9f8e4d92a54ac3e4af22cafc] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=36.6 K 2023-04-27 01:53:59,957 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 77bd45e707cf45f8b88e8680b51c39a5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682560438333 2023-04-27 01:53:59,957 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b7f03d2bf9b54143bc500bb34579bac1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682560438333 2023-04-27 01:53:59,957 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9a2f713ed879442ebda4224af1ef3e03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1682560438492 2023-04-27 01:53:59,957 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 55b5812310a24706bcc5d491dcf0c6b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1682560438492 2023-04-27 01:53:59,958 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 254aa9de247540ceb0c04426d0a1f856, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1682560439123 2023-04-27 01:53:59,959 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 351f83cb9f8e4d92a54ac3e4af22cafc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1682560439123 2023-04-27 01:53:59,967 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:59,974 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:53:59,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:53:59,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:53:59,982 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:53:59,982 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:53:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:53:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:53:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:53:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:53:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:53:59,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:53:59,986 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:53:59,986 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:53:59,986 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:53:59,986 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:53:59,986 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:53:59,987 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/6ebc22518cad430abe050c551873824b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ebc22518cad430abe050c551873824b 2023-04-27 01:53:59,993 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into 6ebc22518cad430abe050c551873824b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:53:59,993 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:53:59,993 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=13, startTime=1682560439955; duration=0sec 2023-04-27 01:53:59,994 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:53:59,994 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:53:59,995 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:53:59,995 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:53:59,995 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:53:59,995 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7104096c4aa142d4afb7c33abd1c594f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21f58f0aa53e453ab6fa7e75a50a733f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/eeba3c1bb4e846d182a29508f729c2f4] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=36.6 K 2023-04-27 01:53:59,996 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7104096c4aa142d4afb7c33abd1c594f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682560438333 2023-04-27 01:53:59,997 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 21f58f0aa53e453ab6fa7e75a50a733f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1682560438492 2023-04-27 01:53:59,997 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting eeba3c1bb4e846d182a29508f729c2f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1682560439123 2023-04-27 01:54:00,005 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:00,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:00,008 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:00,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:00,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:00,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:00,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:00,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:00,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:00,020 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,020 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:00,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:00,020 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:00,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:00,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,023 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560500026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560500026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,051 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:00,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:00,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:00,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:00,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:00,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:00,051 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:00,051 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:00,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:00,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:00,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560500129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560500129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,152 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:00,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:00,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:00,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:00,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:00,155 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:00,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:00,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:00,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:00,156 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,156 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,212 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:00,212 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:00,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:00,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:00,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:00,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:00,214 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:00,214 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:00,214 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:00,214 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:00,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560500235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,243 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:00,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560500243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:00,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47b38673[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:00,246 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:00,247 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560500337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560500337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,382 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,382 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:00,382 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:00,382 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:00,382 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:00,385 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/65b622c2960e40be9ab83f7537f5bbc1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/65b622c2960e40be9ab83f7537f5bbc1 2023-04-27 01:54:00,392 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 65b622c2960e40be9ab83f7537f5bbc1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:00,392 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:00,392 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=13, startTime=1682560439954; duration=0sec 2023-04-27 01:54:00,393 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:00,397 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:00,401 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:00,401 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:00,401 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:00,401 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:00,401 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:00,413 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,413 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,414 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:00,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:00,415 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:00,419 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=812 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/c92e5118778b4752a2def101f06e0e58 2023-04-27 01:54:00,424 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/7a3da56a5e1b40b282c67ee0e847cb82 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7a3da56a5e1b40b282c67ee0e847cb82 2023-04-27 01:54:00,433 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 7a3da56a5e1b40b282c67ee0e847cb82(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:00,433 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:00,433 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=13, startTime=1682560439955; duration=0sec 2023-04-27 01:54:00,433 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:00,466 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,466 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:00,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:00,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:00,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:00,466 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:00,466 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:00,466 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:00,466 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,467 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,467 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,467 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,467 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:00,467 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:00,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:00,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=812 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/c813d87c79d64fdf9dbe008dcf1fd024 2023-04-27 01:54:00,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=812 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/8827180a14a0427f84063201e48391a3 2023-04-27 01:54:00,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/c92e5118778b4752a2def101f06e0e58 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e5118778b4752a2def101f06e0e58 2023-04-27 01:54:00,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e5118778b4752a2def101f06e0e58, entries=150, sequenceid=812, filesize=11.9 K 2023-04-27 01:54:00,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/c813d87c79d64fdf9dbe008dcf1fd024 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/c813d87c79d64fdf9dbe008dcf1fd024 2023-04-27 01:54:00,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/c813d87c79d64fdf9dbe008dcf1fd024, entries=150, sequenceid=812, filesize=11.9 K 2023-04-27 01:54:00,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/8827180a14a0427f84063201e48391a3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/8827180a14a0427f84063201e48391a3 2023-04-27 01:54:00,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/8827180a14a0427f84063201e48391a3, entries=150, sequenceid=812, filesize=11.9 K 2023-04-27 01:54:00,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 4931dc2885b7c715265d908d4b978905 in 538ms, sequenceid=812, compaction requested=false 2023-04-27 01:54:00,546 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:00,567 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:00,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:00,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:00,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:00,572 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:00,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:00,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:00,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:00,573 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,573 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:00,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:00,575 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:00,575 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,576 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:00,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:00,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:00,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:00,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:00,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:00,580 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:00,634 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/290c14b92d624ccd8bbd86bde913b1c2 2023-04-27 01:54:00,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:00,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:00,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560500676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560500677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560500782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560500782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560500984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:00,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:00,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560500984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,054 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/a5c6f21dfbb64b4490db68150cb242df 2023-04-27 01:54:01,071 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/7783580ba1ed4cad8ffe4ebfc640f8f5 2023-04-27 01:54:01,078 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/290c14b92d624ccd8bbd86bde913b1c2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/290c14b92d624ccd8bbd86bde913b1c2 2023-04-27 01:54:01,084 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/290c14b92d624ccd8bbd86bde913b1c2, entries=150, sequenceid=827, filesize=11.9 K 2023-04-27 01:54:01,085 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/a5c6f21dfbb64b4490db68150cb242df as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a5c6f21dfbb64b4490db68150cb242df 2023-04-27 01:54:01,092 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a5c6f21dfbb64b4490db68150cb242df, entries=150, sequenceid=827, filesize=11.9 K 2023-04-27 01:54:01,093 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/7783580ba1ed4cad8ffe4ebfc640f8f5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7783580ba1ed4cad8ffe4ebfc640f8f5 2023-04-27 01:54:01,098 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7783580ba1ed4cad8ffe4ebfc640f8f5, entries=150, sequenceid=827, filesize=11.9 K 2023-04-27 01:54:01,099 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4931dc2885b7c715265d908d4b978905 in 519ms, sequenceid=827, compaction requested=true 2023-04-27 01:54:01,099 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:01,099 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:01,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:01,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:01,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,107 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,107 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:01,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6049864e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:01,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:01,111 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,111 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:01,113 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:01,118 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:01,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:01,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:01,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:01,118 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:01,120 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,120 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,120 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,124 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,124 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,124 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,125 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,125 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:01,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:01,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,125 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:01,125 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:01,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:01,226 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:01,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:01,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:01,231 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:01,231 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,231 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:01,231 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:01,232 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,232 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,233 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,233 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,234 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,234 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:01,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:01,235 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:01,240 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,240 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:01,240 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:54:01,240 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:01,241 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,241 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:01,241 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,241 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:01,241 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:01,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:01,261 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=846 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/bfbf93be5b524718bb779fb110477314 2023-04-27 01:54:01,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560501268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560501269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,286 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=846 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/f2097f804b184a7e881dfaa8a797a46e 2023-04-27 01:54:01,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560501286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560501288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,312 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=846 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/38f460d65f1a47318703b7d067fe9ba1 2023-04-27 01:54:01,318 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/bfbf93be5b524718bb779fb110477314 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/bfbf93be5b524718bb779fb110477314 2023-04-27 01:54:01,326 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/bfbf93be5b524718bb779fb110477314, entries=150, sequenceid=846, filesize=11.9 K 2023-04-27 01:54:01,327 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/f2097f804b184a7e881dfaa8a797a46e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/f2097f804b184a7e881dfaa8a797a46e 2023-04-27 01:54:01,339 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/f2097f804b184a7e881dfaa8a797a46e, entries=150, sequenceid=846, filesize=11.9 K 2023-04-27 01:54:01,342 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/38f460d65f1a47318703b7d067fe9ba1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/38f460d65f1a47318703b7d067fe9ba1 2023-04-27 01:54:01,350 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/38f460d65f1a47318703b7d067fe9ba1, entries=150, sequenceid=846, filesize=11.9 K 2023-04-27 01:54:01,351 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4931dc2885b7c715265d908d4b978905 in 111ms, sequenceid=846, compaction requested=true 2023-04-27 01:54:01,351 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:01,351 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:01,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:01,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:01,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:01,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,353 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,354 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,354 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:01,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2742cd77[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:01,355 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:01,355 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,356 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:01,357 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,357 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:01,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:01,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:01,361 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:01,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:01,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:01,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:01,361 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:01,363 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,363 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,364 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,364 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,368 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,370 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:01,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:01,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,370 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:01,370 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:01,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:01,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:54:01,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:01,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:01,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:01,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/7305c0adc7514a9d82a7df014ff6e509 2023-04-27 01:54:01,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560501408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560501408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,471 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:01,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:01,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:01,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:01,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:01,474 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:01,474 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,474 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:01,474 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:01,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,476 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,476 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,476 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,476 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:01,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:01,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:01,483 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:01,483 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:01,483 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,483 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:01,484 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:01,484 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:01,485 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:01,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a40a9e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:01,491 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:01,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,494 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:01,494 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,494 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:01,503 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:01,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:01,507 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:01,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:01,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:01,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:01,507 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:01,509 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,509 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,509 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,510 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,517 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:01,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560501512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560501512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:01,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,519 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:01,519 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:01,619 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:01,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:01,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:01,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:01,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:01,624 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:01,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:01,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:01,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,624 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,626 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,626 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,626 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,626 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:01,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:01,628 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:01,628 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,628 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:01,628 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:01,628 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:01,628 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,628 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:01,628 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:01,628 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,632 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:01,632 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:01,632 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,632 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:01,632 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:01,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,635 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,635 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137f9de7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:01,636 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:01,636 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,637 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:01,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:01,639 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:01,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:01,643 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:01,643 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:01,643 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:01,643 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:01,643 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:01,645 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,645 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,645 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,646 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,648 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:01,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:01,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,651 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,651 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,652 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:01,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,652 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560501719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560501722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,752 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:01,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:01,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:01,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:01,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:01,757 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:01,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:01,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:01,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,759 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,759 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,759 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,759 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:01,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:01,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:01,762 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,763 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:01,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f281230[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:01,768 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:01,768 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:01,771 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,771 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:01,771 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:01,771 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:01,773 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:01,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:01,777 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:01,777 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:01,777 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:01,778 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:01,778 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:01,778 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:01,778 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,778 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:01,780 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,780 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,780 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:01,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,782 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:01,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:01,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:01,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:01,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,786 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:01,786 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:01,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:01,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:01,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560501789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:01,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560501789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,803 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/883ae10c2eef48a38828416149405223 2023-04-27 01:54:01,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/7f9dac69cc9f47978ba9044630614b06 2023-04-27 01:54:01,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/7305c0adc7514a9d82a7df014ff6e509 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/7305c0adc7514a9d82a7df014ff6e509 2023-04-27 01:54:01,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/7305c0adc7514a9d82a7df014ff6e509, entries=250, sequenceid=861, filesize=16.7 K 2023-04-27 01:54:01,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/883ae10c2eef48a38828416149405223 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/883ae10c2eef48a38828416149405223 2023-04-27 01:54:01,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/883ae10c2eef48a38828416149405223, entries=150, sequenceid=861, filesize=11.9 K 2023-04-27 01:54:01,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/7f9dac69cc9f47978ba9044630614b06 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7f9dac69cc9f47978ba9044630614b06 2023-04-27 01:54:01,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7f9dac69cc9f47978ba9044630614b06, entries=150, sequenceid=861, filesize=11.9 K 2023-04-27 01:54:01,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4931dc2885b7c715265d908d4b978905 in 478ms, sequenceid=861, compaction requested=true 2023-04-27 01:54:01,856 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:01,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:01,857 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 01:54:01,857 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-27 01:54:01,857 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:01,857 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:01,860 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66798 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:01,860 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:01,860 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/A is initiating minor compaction (all files) 2023-04-27 01:54:01,860 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/C is initiating minor compaction (all files) 2023-04-27 01:54:01,860 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/A in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,860 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/C in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,861 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/65b622c2960e40be9ab83f7537f5bbc1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e5118778b4752a2def101f06e0e58, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/290c14b92d624ccd8bbd86bde913b1c2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/bfbf93be5b524718bb779fb110477314, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/7305c0adc7514a9d82a7df014ff6e509] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=65.2 K 2023-04-27 01:54:01,861 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7a3da56a5e1b40b282c67ee0e847cb82, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/8827180a14a0427f84063201e48391a3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7783580ba1ed4cad8ffe4ebfc640f8f5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/38f460d65f1a47318703b7d067fe9ba1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7f9dac69cc9f47978ba9044630614b06] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=60.5 K 2023-04-27 01:54:01,861 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 65b622c2960e40be9ab83f7537f5bbc1, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1682560439123 2023-04-27 01:54:01,861 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7a3da56a5e1b40b282c67ee0e847cb82, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1682560439123 2023-04-27 01:54:01,862 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c92e5118778b4752a2def101f06e0e58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1682560439892 2023-04-27 01:54:01,862 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8827180a14a0427f84063201e48391a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1682560439892 2023-04-27 01:54:01,863 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 290c14b92d624ccd8bbd86bde913b1c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682560440018 2023-04-27 01:54:01,864 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7783580ba1ed4cad8ffe4ebfc640f8f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682560440018 2023-04-27 01:54:01,864 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting bfbf93be5b524718bb779fb110477314, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=846, earliestPutTs=1682560440664 2023-04-27 01:54:01,864 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 38f460d65f1a47318703b7d067fe9ba1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=846, earliestPutTs=1682560440664 2023-04-27 01:54:01,865 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 7305c0adc7514a9d82a7df014ff6e509, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1682560441258 2023-04-27 01:54:01,866 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7f9dac69cc9f47978ba9044630614b06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1682560441258 2023-04-27 01:54:01,886 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:01,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:01,891 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#A#compaction#102 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:01,891 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#C#compaction#103 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:01,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:01,896 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:01,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:01,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:01,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:01,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:01,900 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,901 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:01,901 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:01,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:01,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:01,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:01,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:01,910 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:01,910 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:01,910 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:01,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:01,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:01,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:01,916 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 01:54:01,917 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/9693dab8cad9473781b632c0c4ad6889 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9693dab8cad9473781b632c0c4ad6889 2023-04-27 01:54:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:01,928 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/19543faf6fdd4f35abd3e78e0bf1fdc3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/19543faf6fdd4f35abd3e78e0bf1fdc3 2023-04-27 01:54:01,935 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4931dc2885b7c715265d908d4b978905/C of 4931dc2885b7c715265d908d4b978905 into 19543faf6fdd4f35abd3e78e0bf1fdc3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:01,935 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:01,935 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/C, priority=11, startTime=1682560441857; duration=0sec 2023-04-27 01:54:01,935 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:01,935 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:01,936 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4931dc2885b7c715265d908d4b978905/A of 4931dc2885b7c715265d908d4b978905 into 9693dab8cad9473781b632c0c4ad6889(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:01,936 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:01,936 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/A, priority=11, startTime=1682560441856; duration=0sec 2023-04-27 01:54:01,936 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:01,937 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:01,937 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 4931dc2885b7c715265d908d4b978905/B is initiating minor compaction (all files) 2023-04-27 01:54:01,937 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4931dc2885b7c715265d908d4b978905/B in TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:01,937 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ebc22518cad430abe050c551873824b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/c813d87c79d64fdf9dbe008dcf1fd024, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a5c6f21dfbb64b4490db68150cb242df, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/f2097f804b184a7e881dfaa8a797a46e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/883ae10c2eef48a38828416149405223] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp, totalSize=60.5 K 2023-04-27 01:54:01,938 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6ebc22518cad430abe050c551873824b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=791, earliestPutTs=1682560439123 2023-04-27 01:54:01,938 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c813d87c79d64fdf9dbe008dcf1fd024, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=812, earliestPutTs=1682560439892 2023-04-27 01:54:01,939 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting a5c6f21dfbb64b4490db68150cb242df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682560440018 2023-04-27 01:54:01,939 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f2097f804b184a7e881dfaa8a797a46e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=846, earliestPutTs=1682560440664 2023-04-27 01:54:01,940 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 883ae10c2eef48a38828416149405223, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1682560441258 2023-04-27 01:54:01,960 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=878 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/42e8add1810543ff9977bd2772ed160f 2023-04-27 01:54:01,962 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 4931dc2885b7c715265d908d4b978905#B#compaction#105 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:01,989 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=878 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/3768bdd4df4642c2a35fc9218e555e68 2023-04-27 01:54:02,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:02,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:02,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560502043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560502043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560502145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560502146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560502348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560502349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,378 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/091e340548414d6388209d65f6bffd58 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/091e340548414d6388209d65f6bffd58 2023-04-27 01:54:02,383 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4931dc2885b7c715265d908d4b978905/B of 4931dc2885b7c715265d908d4b978905 into 091e340548414d6388209d65f6bffd58(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:02,383 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:02,383 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., storeName=4931dc2885b7c715265d908d4b978905/B, priority=11, startTime=1682560441856; duration=0sec 2023-04-27 01:54:02,383 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:02,402 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=878 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/ebffe6c2cdbb42d7895fd54f8473a4c7 2023-04-27 01:54:02,407 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/42e8add1810543ff9977bd2772ed160f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/42e8add1810543ff9977bd2772ed160f 2023-04-27 01:54:02,412 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/42e8add1810543ff9977bd2772ed160f, entries=150, sequenceid=878, filesize=11.9 K 2023-04-27 01:54:02,412 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/3768bdd4df4642c2a35fc9218e555e68 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/3768bdd4df4642c2a35fc9218e555e68 2023-04-27 01:54:02,417 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/3768bdd4df4642c2a35fc9218e555e68, entries=150, sequenceid=878, filesize=11.9 K 2023-04-27 01:54:02,418 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/ebffe6c2cdbb42d7895fd54f8473a4c7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/ebffe6c2cdbb42d7895fd54f8473a4c7 2023-04-27 01:54:02,423 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/ebffe6c2cdbb42d7895fd54f8473a4c7, entries=150, sequenceid=878, filesize=11.9 K 2023-04-27 01:54:02,424 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4931dc2885b7c715265d908d4b978905 in 508ms, sequenceid=878, compaction requested=false 2023-04-27 01:54:02,424 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:02,424 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:02,427 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,427 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:02,427 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:02,427 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e3049d7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:02,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:02,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:02,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:02,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:02,437 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:02,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:02,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:02,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:02,437 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:02,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:02,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:02,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:02,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:02,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:02,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:02,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:02,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:02,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:02,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:02,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:02,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:02,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:02,553 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:02,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:02,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:02,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:02,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:02,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:02,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:02,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:02,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:02,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:02,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:02,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:02,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:02,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:02,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:02,558 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:02,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:02,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:02,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:02,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560502685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560502686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41180 deadline: 1682560502769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,774 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:54:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560502789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560502792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41206 deadline: 1682560502799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1682560502799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,991 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/85e45a85f9b849ad8491febf5d61b2de 2023-04-27 01:54:02,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560502992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:02,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:02,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560502997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,015 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/91a086601c3148a69e04643452ae111f 2023-04-27 01:54:03,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:03,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41192 deadline: 1682560503297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:03,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41128 deadline: 1682560503299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,428 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/88f67b3a06be4d64b7f4f8746f984d35 2023-04-27 01:54:03,433 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/85e45a85f9b849ad8491febf5d61b2de as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85e45a85f9b849ad8491febf5d61b2de 2023-04-27 01:54:03,437 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85e45a85f9b849ad8491febf5d61b2de, entries=150, sequenceid=896, filesize=11.9 K 2023-04-27 01:54:03,438 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/91a086601c3148a69e04643452ae111f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/91a086601c3148a69e04643452ae111f 2023-04-27 01:54:03,444 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/91a086601c3148a69e04643452ae111f, entries=150, sequenceid=896, filesize=11.9 K 2023-04-27 01:54:03,445 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/88f67b3a06be4d64b7f4f8746f984d35 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/88f67b3a06be4d64b7f4f8746f984d35 2023-04-27 01:54:03,450 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/88f67b3a06be4d64b7f4f8746f984d35, entries=150, sequenceid=896, filesize=11.9 K 2023-04-27 01:54:03,451 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4931dc2885b7c715265d908d4b978905 in 893ms, sequenceid=896, compaction requested=true 2023-04-27 01:54:03,451 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:03,451 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:03,451 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:03,451 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:03,452 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,452 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:03,452 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:03,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,454 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:03,454 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,454 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:03,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:03,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66a84275[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:03,456 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:03,456 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,458 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:03,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:03,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:03,459 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,459 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:03,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:03,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,460 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:03,460 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:03,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:03,462 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:03,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:03,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:03,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:03,462 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:03,464 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,464 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:03,464 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:03,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:03,465 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:03,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:03,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:03,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:03,472 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:03,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:03,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:03,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:03,473 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:03,473 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:03,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:03,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:03,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:03,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:03,573 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:03,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:03,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:03,576 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:03,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:03,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:03,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:03,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:03,580 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:03,580 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:03,580 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:03,580 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:03,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:03,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:03,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:03,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:03,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:03,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:03,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:03,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:03,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:03,582 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:03,582 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. started... 2023-04-27 01:54:03,582 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 01:54:03,582 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:03,582 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:03,582 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:03,582 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:03,582 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:03,583 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:03,591 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75f5a5e6 to 127.0.0.1:52731 2023-04-27 01:54:03,591 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:03,592 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x726fb449 to 127.0.0.1:52731 2023-04-27 01:54:03,592 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:03,596 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71f9f426 to 127.0.0.1:52731 2023-04-27 01:54:03,596 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ae6c42b to 127.0.0.1:52731 2023-04-27 01:54:03,596 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:03,596 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:03,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:03,801 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. as already flushing 2023-04-27 01:54:03,801 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b41e306 to 127.0.0.1:52731 2023-04-27 01:54:03,801 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:03,803 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e8196e0 to 127.0.0.1:52731 2023-04-27 01:54:03,803 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:04,001 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=913 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/c92e7c57320648d48179d323e7e852a2 2023-04-27 01:54:04,412 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=913 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/7e1f65c7cf08493cba300f058c6d9a44 2023-04-27 01:54:04,809 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377933e4 to 127.0.0.1:52731 2023-04-27 01:54:04,809 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:04,812 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2411a862 to 127.0.0.1:52731 2023-04-27 01:54:04,812 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:04,823 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=913 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/f95affa1f3834567bb6b82d4bb983178 2023-04-27 01:54:04,828 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/c92e7c57320648d48179d323e7e852a2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e7c57320648d48179d323e7e852a2 2023-04-27 01:54:04,832 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e7c57320648d48179d323e7e852a2, entries=150, sequenceid=913, filesize=11.9 K 2023-04-27 01:54:04,832 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/7e1f65c7cf08493cba300f058c6d9a44 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/7e1f65c7cf08493cba300f058c6d9a44 2023-04-27 01:54:04,836 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/7e1f65c7cf08493cba300f058c6d9a44, entries=150, sequenceid=913, filesize=11.9 K 2023-04-27 01:54:04,837 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/f95affa1f3834567bb6b82d4bb983178 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/f95affa1f3834567bb6b82d4bb983178 2023-04-27 01:54:04,841 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/f95affa1f3834567bb6b82d4bb983178, entries=150, sequenceid=913, filesize=11.9 K 2023-04-27 01:54:04,842 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=26.84 KB/27480 for 4931dc2885b7c715265d908d4b978905 in 1260ms, sequenceid=913, compaction requested=true 2023-04-27 01:54:04,842 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:04,842 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:04,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:04,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:04,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:04,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:04,844 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,844 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:04,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:04,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:04,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d77be2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:04,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:04,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:04,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:04,852 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:04,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,852 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:04,853 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:04,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:04,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:04,854 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:04,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:04,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:04,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:04,855 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:04,861 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:04,861 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:04,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:04,861 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:04,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:04,861 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,862 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,865 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:04,865 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:04,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:04,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:04,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:04,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:04,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:04,866 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:04,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:04,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:04,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,866 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:04,866 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:04,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:04,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:04,966 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:04,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:12,824 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ca4a062 to 127.0.0.1:52731 2023-04-27 01:54:12,825 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 193 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 162 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 3951 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 3908 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1758 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 5267 rows 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1723 2023-04-27 01:54:12,825 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 5160 rows 2023-04-27 01:54:12,826 INFO [Listener at localhost/42495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 01:54:12,826 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36d9637e to 127.0.0.1:52731 2023-04-27 01:54:12,826 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:12,830 INFO [Listener at localhost/42495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 01:54:12,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 01:54:12,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:12,845 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560452845"}]},"ts":"1682560452845"} 2023-04-27 01:54:12,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 01:54:12,846 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 01:54:12,931 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 01:54:12,933 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4931dc2885b7c715265d908d4b978905, UNASSIGN}] 2023-04-27 01:54:12,935 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4931dc2885b7c715265d908d4b978905, UNASSIGN 2023-04-27 01:54:12,935 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=4931dc2885b7c715265d908d4b978905, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:12,935 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560452935"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560452935"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560452935"}]},"ts":"1682560452935"} 2023-04-27 01:54:12,937 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:54:12,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 01:54:13,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:13,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4931dc2885b7c715265d908d4b978905, disabling compactions & flushes 2023-04-27 01:54:13,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:13,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:13,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. after waiting 0 ms 2023-04-27 01:54:13,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:13,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4931dc2885b7c715265d908d4b978905 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-27 01:54:13,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=A 2023-04-27 01:54:13,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:13,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=B 2023-04-27 01:54:13,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:13,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4931dc2885b7c715265d908d4b978905, store=C 2023-04-27 01:54:13,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:13,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 01:54:13,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 01:54:13,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=921 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/66a4559ae7ad4e64a642f2d67d021d18 2023-04-27 01:54:13,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=921 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/6ede3c31d0cc4149ab78f6e4f8de5ff9 2023-04-27 01:54:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 01:54:14,331 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=921 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/82eadcd5ddea43da960a933640f5d615 2023-04-27 01:54:14,335 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/A/66a4559ae7ad4e64a642f2d67d021d18 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/66a4559ae7ad4e64a642f2d67d021d18 2023-04-27 01:54:14,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/66a4559ae7ad4e64a642f2d67d021d18, entries=150, sequenceid=921, filesize=11.9 K 2023-04-27 01:54:14,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/B/6ede3c31d0cc4149ab78f6e4f8de5ff9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ede3c31d0cc4149ab78f6e4f8de5ff9 2023-04-27 01:54:14,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ede3c31d0cc4149ab78f6e4f8de5ff9, entries=150, sequenceid=921, filesize=11.9 K 2023-04-27 01:54:14,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/.tmp/C/82eadcd5ddea43da960a933640f5d615 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/82eadcd5ddea43da960a933640f5d615 2023-04-27 01:54:14,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/82eadcd5ddea43da960a933640f5d615, entries=150, sequenceid=921, filesize=11.9 K 2023-04-27 01:54:14,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 4931dc2885b7c715265d908d4b978905 in 1252ms, sequenceid=921, compaction requested=true 2023-04-27 01:54:14,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/fbdbca1527ec45d0a9e63af4931d80c6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/95a5a49dfa904007bfd3652b63a9b0b0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3221cf3739e24f4db7e4657ad341b000, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/f78c73d14c4f406b9b5bab2d1423e3a3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/163ccd0894464c2baafa5eaa64edd889, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/395b29307238497ba7945ede954d18f8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/16de9f877caa4463afc1a48d6dbedc9c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e17804849541494b930227b7e66c20c5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/17a3c918d9ff45daa630b02f46bc797b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/5270dfe6db28434ab1d5f9fc42481038, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e2eb0fe0cfa045c894f534ce8ee8ede7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85199c1ad627440ba207ed0d44123d65, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2c1ed7dca392474884ce0a0a76aad176, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/573d660bc30044d8b6def4ad8d189bfc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/942092d941f940118284729dabb43a6b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2e97386bc79b4dfbad4cc2651a92933e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/ee1de37a00144ee086f5cbf2040a0aae, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/56b7f525217e412892326375287e6942, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4dc48826045945d792071b532a6ed473, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/30ad44c6dd034a22a09e52088bf48d9b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3932051de581419bb76203cb2f6b08ab, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/12c18a93e6724fbdbbfcaa78656759b6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4ee47c9e1b8c4e6a82aabbeef104834e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2ba569db7e0b45b6866b8b5ad64439cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/88284bf7db164b13bceff1cd623a2e9a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/77bd45e707cf45f8b88e8680b51c39a5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3799d4e39ad640fd99898709502d1f8a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9a2f713ed879442ebda4224af1ef3e03, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/254aa9de247540ceb0c04426d0a1f856, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/65b622c2960e40be9ab83f7537f5bbc1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e5118778b4752a2def101f06e0e58, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/290c14b92d624ccd8bbd86bde913b1c2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/bfbf93be5b524718bb779fb110477314, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/7305c0adc7514a9d82a7df014ff6e509] to archive 2023-04-27 01:54:14,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:54:14,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/fbdbca1527ec45d0a9e63af4931d80c6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/fbdbca1527ec45d0a9e63af4931d80c6 2023-04-27 01:54:14,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/95a5a49dfa904007bfd3652b63a9b0b0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/95a5a49dfa904007bfd3652b63a9b0b0 2023-04-27 01:54:14,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3221cf3739e24f4db7e4657ad341b000 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3221cf3739e24f4db7e4657ad341b000 2023-04-27 01:54:14,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/f78c73d14c4f406b9b5bab2d1423e3a3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/f78c73d14c4f406b9b5bab2d1423e3a3 2023-04-27 01:54:14,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/163ccd0894464c2baafa5eaa64edd889 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/163ccd0894464c2baafa5eaa64edd889 2023-04-27 01:54:14,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/395b29307238497ba7945ede954d18f8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/395b29307238497ba7945ede954d18f8 2023-04-27 01:54:14,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/16de9f877caa4463afc1a48d6dbedc9c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/16de9f877caa4463afc1a48d6dbedc9c 2023-04-27 01:54:14,384 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e17804849541494b930227b7e66c20c5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e17804849541494b930227b7e66c20c5 2023-04-27 01:54:14,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/17a3c918d9ff45daa630b02f46bc797b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/17a3c918d9ff45daa630b02f46bc797b 2023-04-27 01:54:14,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/5270dfe6db28434ab1d5f9fc42481038 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/5270dfe6db28434ab1d5f9fc42481038 2023-04-27 01:54:14,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e2eb0fe0cfa045c894f534ce8ee8ede7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/e2eb0fe0cfa045c894f534ce8ee8ede7 2023-04-27 01:54:14,389 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85199c1ad627440ba207ed0d44123d65 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85199c1ad627440ba207ed0d44123d65 2023-04-27 01:54:14,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2c1ed7dca392474884ce0a0a76aad176 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2c1ed7dca392474884ce0a0a76aad176 2023-04-27 01:54:14,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/573d660bc30044d8b6def4ad8d189bfc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/573d660bc30044d8b6def4ad8d189bfc 2023-04-27 01:54:14,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/942092d941f940118284729dabb43a6b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/942092d941f940118284729dabb43a6b 2023-04-27 01:54:14,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2e97386bc79b4dfbad4cc2651a92933e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2e97386bc79b4dfbad4cc2651a92933e 2023-04-27 01:54:14,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/ee1de37a00144ee086f5cbf2040a0aae to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/ee1de37a00144ee086f5cbf2040a0aae 2023-04-27 01:54:14,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/56b7f525217e412892326375287e6942 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/56b7f525217e412892326375287e6942 2023-04-27 01:54:14,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4dc48826045945d792071b532a6ed473 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4dc48826045945d792071b532a6ed473 2023-04-27 01:54:14,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/30ad44c6dd034a22a09e52088bf48d9b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/30ad44c6dd034a22a09e52088bf48d9b 2023-04-27 01:54:14,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3932051de581419bb76203cb2f6b08ab to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3932051de581419bb76203cb2f6b08ab 2023-04-27 01:54:14,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/12c18a93e6724fbdbbfcaa78656759b6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/12c18a93e6724fbdbbfcaa78656759b6 2023-04-27 01:54:14,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4ee47c9e1b8c4e6a82aabbeef104834e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/4ee47c9e1b8c4e6a82aabbeef104834e 2023-04-27 01:54:14,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2ba569db7e0b45b6866b8b5ad64439cc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/2ba569db7e0b45b6866b8b5ad64439cc 2023-04-27 01:54:14,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/88284bf7db164b13bceff1cd623a2e9a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/88284bf7db164b13bceff1cd623a2e9a 2023-04-27 01:54:14,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/77bd45e707cf45f8b88e8680b51c39a5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/77bd45e707cf45f8b88e8680b51c39a5 2023-04-27 01:54:14,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3799d4e39ad640fd99898709502d1f8a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/3799d4e39ad640fd99898709502d1f8a 2023-04-27 01:54:14,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9a2f713ed879442ebda4224af1ef3e03 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9a2f713ed879442ebda4224af1ef3e03 2023-04-27 01:54:14,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/254aa9de247540ceb0c04426d0a1f856 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/254aa9de247540ceb0c04426d0a1f856 2023-04-27 01:54:14,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/65b622c2960e40be9ab83f7537f5bbc1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/65b622c2960e40be9ab83f7537f5bbc1 2023-04-27 01:54:14,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e5118778b4752a2def101f06e0e58 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e5118778b4752a2def101f06e0e58 2023-04-27 01:54:14,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/290c14b92d624ccd8bbd86bde913b1c2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/290c14b92d624ccd8bbd86bde913b1c2 2023-04-27 01:54:14,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/bfbf93be5b524718bb779fb110477314 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/bfbf93be5b524718bb779fb110477314 2023-04-27 01:54:14,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/7305c0adc7514a9d82a7df014ff6e509 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/7305c0adc7514a9d82a7df014ff6e509 2023-04-27 01:54:14,438 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2c2f762b237c48bd8c7474563c40c104, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351e1f1c319845459a5a3d2a9a97dd1e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/17da083b0a9e4899be8ab894bdf4fe76, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b18a1ef7375e4c86857edb7ebe2ac5e2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/480288958a4d428faaed3decccd7ded8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d3fc9aaf0969441a952810a323f92b8f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/dea4d02b46234615800c74d6a335a04b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/72eabe9719a54ebe99c98c8209a2d87b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/994107157dac47dcaec1ead8f81ac326, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/175b277e76a642dc9d37d5f864339d12, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/e80f4690e8894aa5a976c1f544253499, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d146babeb67e4ca2af53225307df6244, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a30625908115428cb211c283cde737fb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/1efdb21e9ed1494f9609b317417e3497, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/8aa99d20895d4bde868eea19599f90e8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/991792dfcfb34893a0b02fccc409ff69, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/9a9dd9eb814c4ef7b7c9b2b9f7f67cdf, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d47002afc7f641a78407c95e23eea2ae, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b525a104a8a84ed4a37d15a59ffc2c7d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/86519a386c434acc8c8aecb1d7b34824, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d63890185ada486fbe22244d1f722d6e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/0c635d15740b4f51a1b40fde59a505bd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/356976fd419a46e4bb9c34f99de47fc5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/abd9768649e94478b12d5c3b20686ec7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b7f03d2bf9b54143bc500bb34579bac1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/866cb85b326e4200a5af7817c0045ee9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/55b5812310a24706bcc5d491dcf0c6b3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ebc22518cad430abe050c551873824b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351f83cb9f8e4d92a54ac3e4af22cafc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/c813d87c79d64fdf9dbe008dcf1fd024, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a5c6f21dfbb64b4490db68150cb242df, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/f2097f804b184a7e881dfaa8a797a46e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/883ae10c2eef48a38828416149405223] to archive 2023-04-27 01:54:14,439 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:54:14,441 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2c2f762b237c48bd8c7474563c40c104 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2c2f762b237c48bd8c7474563c40c104 2023-04-27 01:54:14,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351e1f1c319845459a5a3d2a9a97dd1e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351e1f1c319845459a5a3d2a9a97dd1e 2023-04-27 01:54:14,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/17da083b0a9e4899be8ab894bdf4fe76 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/17da083b0a9e4899be8ab894bdf4fe76 2023-04-27 01:54:14,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b18a1ef7375e4c86857edb7ebe2ac5e2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b18a1ef7375e4c86857edb7ebe2ac5e2 2023-04-27 01:54:14,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/2ae8cbe45f5a46e4a0c47bdc23b0edc5 2023-04-27 01:54:14,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/480288958a4d428faaed3decccd7ded8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/480288958a4d428faaed3decccd7ded8 2023-04-27 01:54:14,447 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d3fc9aaf0969441a952810a323f92b8f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d3fc9aaf0969441a952810a323f92b8f 2023-04-27 01:54:14,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/dea4d02b46234615800c74d6a335a04b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/dea4d02b46234615800c74d6a335a04b 2023-04-27 01:54:14,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/72eabe9719a54ebe99c98c8209a2d87b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/72eabe9719a54ebe99c98c8209a2d87b 2023-04-27 01:54:14,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/994107157dac47dcaec1ead8f81ac326 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/994107157dac47dcaec1ead8f81ac326 2023-04-27 01:54:14,451 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/175b277e76a642dc9d37d5f864339d12 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/175b277e76a642dc9d37d5f864339d12 2023-04-27 01:54:14,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/e80f4690e8894aa5a976c1f544253499 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/e80f4690e8894aa5a976c1f544253499 2023-04-27 01:54:14,453 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d146babeb67e4ca2af53225307df6244 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d146babeb67e4ca2af53225307df6244 2023-04-27 01:54:14,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a30625908115428cb211c283cde737fb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a30625908115428cb211c283cde737fb 2023-04-27 01:54:14,455 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/1efdb21e9ed1494f9609b317417e3497 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/1efdb21e9ed1494f9609b317417e3497 2023-04-27 01:54:14,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/8aa99d20895d4bde868eea19599f90e8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/8aa99d20895d4bde868eea19599f90e8 2023-04-27 01:54:14,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/991792dfcfb34893a0b02fccc409ff69 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/991792dfcfb34893a0b02fccc409ff69 2023-04-27 01:54:14,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/9a9dd9eb814c4ef7b7c9b2b9f7f67cdf to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/9a9dd9eb814c4ef7b7c9b2b9f7f67cdf 2023-04-27 01:54:14,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d47002afc7f641a78407c95e23eea2ae to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d47002afc7f641a78407c95e23eea2ae 2023-04-27 01:54:14,460 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b525a104a8a84ed4a37d15a59ffc2c7d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b525a104a8a84ed4a37d15a59ffc2c7d 2023-04-27 01:54:14,462 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/86519a386c434acc8c8aecb1d7b34824 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/86519a386c434acc8c8aecb1d7b34824 2023-04-27 01:54:14,463 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d63890185ada486fbe22244d1f722d6e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/d63890185ada486fbe22244d1f722d6e 2023-04-27 01:54:14,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/0c635d15740b4f51a1b40fde59a505bd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/0c635d15740b4f51a1b40fde59a505bd 2023-04-27 01:54:14,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/356976fd419a46e4bb9c34f99de47fc5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/356976fd419a46e4bb9c34f99de47fc5 2023-04-27 01:54:14,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/abd9768649e94478b12d5c3b20686ec7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/abd9768649e94478b12d5c3b20686ec7 2023-04-27 01:54:14,467 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b7f03d2bf9b54143bc500bb34579bac1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/b7f03d2bf9b54143bc500bb34579bac1 2023-04-27 01:54:14,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/866cb85b326e4200a5af7817c0045ee9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/866cb85b326e4200a5af7817c0045ee9 2023-04-27 01:54:14,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/55b5812310a24706bcc5d491dcf0c6b3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/55b5812310a24706bcc5d491dcf0c6b3 2023-04-27 01:54:14,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ebc22518cad430abe050c551873824b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ebc22518cad430abe050c551873824b 2023-04-27 01:54:14,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351f83cb9f8e4d92a54ac3e4af22cafc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/351f83cb9f8e4d92a54ac3e4af22cafc 2023-04-27 01:54:14,472 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/c813d87c79d64fdf9dbe008dcf1fd024 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/c813d87c79d64fdf9dbe008dcf1fd024 2023-04-27 01:54:14,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a5c6f21dfbb64b4490db68150cb242df to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/a5c6f21dfbb64b4490db68150cb242df 2023-04-27 01:54:14,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/f2097f804b184a7e881dfaa8a797a46e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/f2097f804b184a7e881dfaa8a797a46e 2023-04-27 01:54:14,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/883ae10c2eef48a38828416149405223 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/883ae10c2eef48a38828416149405223 2023-04-27 01:54:14,479 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/04671756a40749cea17a7e3f3dc6c504, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/854b14cf67c24a47a199526c60fc71a0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d99ab51cd7fd467bb97291fa3e40737a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/9f68edd2ea9e4a6e880f9ccb9b9dd151, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1cdbcf8a6e384c4cac4553c65dfe0d20, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1da99a232f14e579d8e458fb3240727, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6923777d138a494ebaecf05d8b6c0ead, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1a6f8357e004207981e6b1ff7b24154, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0279811906624ae19caf7f22fb279ec2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d5a46e09cdd438bb1f0ff650993189f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c24b5c3f428b49559a1b8460412cd2e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1d84a6d56ce2481e8e3bd61bab10caf6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d2dc821ec12a44c288e67df49a80f451, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/cf1626a20c7f42edbe23d8a5888a5a6e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/80bfb4fb56584b62b2bec7e3870a1fd2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/b33b88156f0a4375bd86e659205a0a76, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/4fbdcd5869d14c539e82454b304f7312, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6ce1dab730f546848203949c3b73da86, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0649f098b0234849bd336d6142e06f6a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21d7ee2094d74e71a9be6ddd6b000f9e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1ea2b1199fb247f094b5333de598ce77, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c03e6d134dcb4f84a9051ca8718d8dc3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/934e3e451c874ceeaa13668b4325137f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d66921f4ad724c6cb9379fc07cc76967, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d9e78a97c6f41a9ad864c570cee2769, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7104096c4aa142d4afb7c33abd1c594f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/11020a3daf3c4b3dbbe1f0b5984bbdc6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21f58f0aa53e453ab6fa7e75a50a733f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7a3da56a5e1b40b282c67ee0e847cb82, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/eeba3c1bb4e846d182a29508f729c2f4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/8827180a14a0427f84063201e48391a3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7783580ba1ed4cad8ffe4ebfc640f8f5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/38f460d65f1a47318703b7d067fe9ba1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7f9dac69cc9f47978ba9044630614b06] to archive 2023-04-27 01:54:14,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:54:14,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/04671756a40749cea17a7e3f3dc6c504 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/04671756a40749cea17a7e3f3dc6c504 2023-04-27 01:54:14,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/854b14cf67c24a47a199526c60fc71a0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/854b14cf67c24a47a199526c60fc71a0 2023-04-27 01:54:14,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d99ab51cd7fd467bb97291fa3e40737a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d99ab51cd7fd467bb97291fa3e40737a 2023-04-27 01:54:14,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/9f68edd2ea9e4a6e880f9ccb9b9dd151 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/9f68edd2ea9e4a6e880f9ccb9b9dd151 2023-04-27 01:54:14,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1cdbcf8a6e384c4cac4553c65dfe0d20 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1cdbcf8a6e384c4cac4553c65dfe0d20 2023-04-27 01:54:14,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1da99a232f14e579d8e458fb3240727 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1da99a232f14e579d8e458fb3240727 2023-04-27 01:54:14,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6923777d138a494ebaecf05d8b6c0ead to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6923777d138a494ebaecf05d8b6c0ead 2023-04-27 01:54:14,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1a6f8357e004207981e6b1ff7b24154 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/a1a6f8357e004207981e6b1ff7b24154 2023-04-27 01:54:14,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0279811906624ae19caf7f22fb279ec2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0279811906624ae19caf7f22fb279ec2 2023-04-27 01:54:14,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d5a46e09cdd438bb1f0ff650993189f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d5a46e09cdd438bb1f0ff650993189f 2023-04-27 01:54:14,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c24b5c3f428b49559a1b8460412cd2e5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c24b5c3f428b49559a1b8460412cd2e5 2023-04-27 01:54:14,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1d84a6d56ce2481e8e3bd61bab10caf6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1d84a6d56ce2481e8e3bd61bab10caf6 2023-04-27 01:54:14,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d2dc821ec12a44c288e67df49a80f451 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d2dc821ec12a44c288e67df49a80f451 2023-04-27 01:54:14,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/cf1626a20c7f42edbe23d8a5888a5a6e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/cf1626a20c7f42edbe23d8a5888a5a6e 2023-04-27 01:54:14,496 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/80bfb4fb56584b62b2bec7e3870a1fd2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/80bfb4fb56584b62b2bec7e3870a1fd2 2023-04-27 01:54:14,497 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/b33b88156f0a4375bd86e659205a0a76 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/b33b88156f0a4375bd86e659205a0a76 2023-04-27 01:54:14,498 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/4fbdcd5869d14c539e82454b304f7312 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/4fbdcd5869d14c539e82454b304f7312 2023-04-27 01:54:14,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6ce1dab730f546848203949c3b73da86 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/6ce1dab730f546848203949c3b73da86 2023-04-27 01:54:14,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0649f098b0234849bd336d6142e06f6a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/0649f098b0234849bd336d6142e06f6a 2023-04-27 01:54:14,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21d7ee2094d74e71a9be6ddd6b000f9e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21d7ee2094d74e71a9be6ddd6b000f9e 2023-04-27 01:54:14,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1ea2b1199fb247f094b5333de598ce77 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/1ea2b1199fb247f094b5333de598ce77 2023-04-27 01:54:14,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c03e6d134dcb4f84a9051ca8718d8dc3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/c03e6d134dcb4f84a9051ca8718d8dc3 2023-04-27 01:54:14,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/934e3e451c874ceeaa13668b4325137f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/934e3e451c874ceeaa13668b4325137f 2023-04-27 01:54:14,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d66921f4ad724c6cb9379fc07cc76967 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/d66921f4ad724c6cb9379fc07cc76967 2023-04-27 01:54:14,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d9e78a97c6f41a9ad864c570cee2769 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/3d9e78a97c6f41a9ad864c570cee2769 2023-04-27 01:54:14,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7104096c4aa142d4afb7c33abd1c594f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7104096c4aa142d4afb7c33abd1c594f 2023-04-27 01:54:14,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/11020a3daf3c4b3dbbe1f0b5984bbdc6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/11020a3daf3c4b3dbbe1f0b5984bbdc6 2023-04-27 01:54:14,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21f58f0aa53e453ab6fa7e75a50a733f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/21f58f0aa53e453ab6fa7e75a50a733f 2023-04-27 01:54:14,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7a3da56a5e1b40b282c67ee0e847cb82 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7a3da56a5e1b40b282c67ee0e847cb82 2023-04-27 01:54:14,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/eeba3c1bb4e846d182a29508f729c2f4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/eeba3c1bb4e846d182a29508f729c2f4 2023-04-27 01:54:14,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/8827180a14a0427f84063201e48391a3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/8827180a14a0427f84063201e48391a3 2023-04-27 01:54:14,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7783580ba1ed4cad8ffe4ebfc640f8f5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7783580ba1ed4cad8ffe4ebfc640f8f5 2023-04-27 01:54:14,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/38f460d65f1a47318703b7d067fe9ba1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/38f460d65f1a47318703b7d067fe9ba1 2023-04-27 01:54:14,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7f9dac69cc9f47978ba9044630614b06 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/7f9dac69cc9f47978ba9044630614b06 2023-04-27 01:54:14,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/recovered.edits/924.seqid, newMaxSeqId=924, maxSeqId=1 2023-04-27 01:54:14,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905. 2023-04-27 01:54:14,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4931dc2885b7c715265d908d4b978905: 2023-04-27 01:54:14,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:14,525 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=4931dc2885b7c715265d908d4b978905, regionState=CLOSED 2023-04-27 01:54:14,525 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560454525"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560454525"}]},"ts":"1682560454525"} 2023-04-27 01:54:14,528 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-27 01:54:14,528 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 4931dc2885b7c715265d908d4b978905, server=jenkins-hbase4.apache.org,38085,1682560417985 in 1.5890 sec 2023-04-27 01:54:14,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-27 01:54:14,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4931dc2885b7c715265d908d4b978905, UNASSIGN in 1.5950 sec 2023-04-27 01:54:14,530 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560454530"}]},"ts":"1682560454530"} 2023-04-27 01:54:14,531 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 01:54:14,533 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 01:54:14,535 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6970 sec 2023-04-27 01:54:14,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 01:54:14,951 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-27 01:54:14,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 01:54:14,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:14,963 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:14,965 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:14,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-27 01:54:14,970 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:14,974 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/recovered.edits] 2023-04-27 01:54:14,977 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/42e8add1810543ff9977bd2772ed160f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/42e8add1810543ff9977bd2772ed160f 2023-04-27 01:54:14,979 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/66a4559ae7ad4e64a642f2d67d021d18 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/66a4559ae7ad4e64a642f2d67d021d18 2023-04-27 01:54:14,980 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85e45a85f9b849ad8491febf5d61b2de to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/85e45a85f9b849ad8491febf5d61b2de 2023-04-27 01:54:14,982 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9693dab8cad9473781b632c0c4ad6889 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/9693dab8cad9473781b632c0c4ad6889 2023-04-27 01:54:14,983 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e7c57320648d48179d323e7e852a2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/A/c92e7c57320648d48179d323e7e852a2 2023-04-27 01:54:14,985 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/091e340548414d6388209d65f6bffd58 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/091e340548414d6388209d65f6bffd58 2023-04-27 01:54:14,987 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/3768bdd4df4642c2a35fc9218e555e68 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/3768bdd4df4642c2a35fc9218e555e68 2023-04-27 01:54:14,988 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ede3c31d0cc4149ab78f6e4f8de5ff9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/6ede3c31d0cc4149ab78f6e4f8de5ff9 2023-04-27 01:54:14,989 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/7e1f65c7cf08493cba300f058c6d9a44 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/7e1f65c7cf08493cba300f058c6d9a44 2023-04-27 01:54:14,990 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/91a086601c3148a69e04643452ae111f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/B/91a086601c3148a69e04643452ae111f 2023-04-27 01:54:14,993 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/19543faf6fdd4f35abd3e78e0bf1fdc3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/19543faf6fdd4f35abd3e78e0bf1fdc3 2023-04-27 01:54:14,994 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/82eadcd5ddea43da960a933640f5d615 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/82eadcd5ddea43da960a933640f5d615 2023-04-27 01:54:14,996 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/88f67b3a06be4d64b7f4f8746f984d35 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/88f67b3a06be4d64b7f4f8746f984d35 2023-04-27 01:54:14,997 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/ebffe6c2cdbb42d7895fd54f8473a4c7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/ebffe6c2cdbb42d7895fd54f8473a4c7 2023-04-27 01:54:14,998 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/f95affa1f3834567bb6b82d4bb983178 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/C/f95affa1f3834567bb6b82d4bb983178 2023-04-27 01:54:15,001 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/recovered.edits/924.seqid to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905/recovered.edits/924.seqid 2023-04-27 01:54:15,002 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/4931dc2885b7c715265d908d4b978905 2023-04-27 01:54:15,002 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:54:15,004 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:15,025 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 01:54:15,068 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 01:54:15,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-27 01:54:15,069 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:15,069 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 01:54:15,069 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682560455069"}]},"ts":"9223372036854775807"} 2023-04-27 01:54:15,072 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 01:54:15,072 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4931dc2885b7c715265d908d4b978905, NAME => 'TestAcidGuarantees,,1682560422331.4931dc2885b7c715265d908d4b978905.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 01:54:15,072 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 01:54:15,072 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682560455072"}]},"ts":"9223372036854775807"} 2023-04-27 01:54:15,074 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 01:54:15,076 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:15,077 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 120 msec 2023-04-27 01:54:15,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-27 01:54:15,270 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-27 01:54:15,287 INFO [Listener at localhost/42495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=229 (was 205) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4ebc138c-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_109255128_15 at /127.0.0.1:47290 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4ebc138c-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4ebc138c-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:38085-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4ebc138c-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=542 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 333) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12068 (was 12717) 2023-04-27 01:54:15,298 INFO [Listener at localhost/42495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229, OpenFileDescriptor=542, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=170, AvailableMemoryMB=12068 2023-04-27 01:54:15,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:54:15,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:54:15,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:15,303 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 01:54:15,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-27 01:54:15,304 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 01:54:15,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 01:54:15,306 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:15,306 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 empty. 2023-04-27 01:54:15,307 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:15,307 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:54:15,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 01:54:15,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 01:54:15,716 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:54:15,718 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => da5b8d348b0d50311fe8eae652165c86, NAME => 'TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp 2023-04-27 01:54:15,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 01:54:16,124 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:54:16,124 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing da5b8d348b0d50311fe8eae652165c86, disabling compactions & flushes 2023-04-27 01:54:16,124 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,124 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,124 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. after waiting 0 ms 2023-04-27 01:54:16,124 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,124 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,124 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:16,127 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 01:54:16,128 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560456127"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560456127"}]},"ts":"1682560456127"} 2023-04-27 01:54:16,129 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 01:54:16,130 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 01:54:16,130 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560456130"}]},"ts":"1682560456130"} 2023-04-27 01:54:16,131 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 01:54:16,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, ASSIGN}] 2023-04-27 01:54:16,135 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, ASSIGN 2023-04-27 01:54:16,135 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=false 2023-04-27 01:54:16,287 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:16,287 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560456286"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560456286"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560456286"}]},"ts":"1682560456286"} 2023-04-27 01:54:16,289 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:54:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 01:54:16,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => da5b8d348b0d50311fe8eae652165c86, NAME => 'TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:54:16,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:54:16,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,446 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,448 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A 2023-04-27 01:54:16,448 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A 2023-04-27 01:54:16,448 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:16,448 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da5b8d348b0d50311fe8eae652165c86 columnFamilyName A 2023-04-27 01:54:16,449 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(310): Store=da5b8d348b0d50311fe8eae652165c86/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:16,449 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,450 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B 2023-04-27 01:54:16,450 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B 2023-04-27 01:54:16,450 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:16,450 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da5b8d348b0d50311fe8eae652165c86 columnFamilyName B 2023-04-27 01:54:16,451 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(310): Store=da5b8d348b0d50311fe8eae652165c86/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:16,451 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,452 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C 2023-04-27 01:54:16,452 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C 2023-04-27 01:54:16,452 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:16,453 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da5b8d348b0d50311fe8eae652165c86 columnFamilyName C 2023-04-27 01:54:16,453 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(310): Store=da5b8d348b0d50311fe8eae652165c86/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:16,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:54:16,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:16,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:54:16,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened da5b8d348b0d50311fe8eae652165c86; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65489047, jitterRate=-0.02413715422153473}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:54:16,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:16,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., pid=18, masterSystemTime=1682560456440 2023-04-27 01:54:16,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:16,462 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:16,463 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560456462"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560456462"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560456462"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560456462"}]},"ts":"1682560456462"} 2023-04-27 01:54:16,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-27 01:54:16,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 in 175 msec 2023-04-27 01:54:16,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-27 01:54:16,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, ASSIGN in 332 msec 2023-04-27 01:54:16,468 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 01:54:16,468 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560456468"}]},"ts":"1682560456468"} 2023-04-27 01:54:16,469 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 01:54:16,471 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 01:54:16,472 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-27 01:54:17,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 01:54:17,409 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-27 01:54:17,411 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x58d22e5a to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:17,419 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4910cda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:17,421 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:17,425 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:17,427 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 01:54:17,429 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 01:54:17,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:54:17,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:54:17,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:17,857 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 01:54:17,858 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:54:17,858 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 01:54:17,862 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-27 01:54:17,874 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, REOPEN/MOVE}] 2023-04-27 01:54:17,875 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, REOPEN/MOVE 2023-04-27 01:54:17,876 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:17,876 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560457876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560457876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560457876"}]},"ts":"1682560457876"} 2023-04-27 01:54:17,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:54:18,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing da5b8d348b0d50311fe8eae652165c86, disabling compactions & flushes 2023-04-27 01:54:18,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. after waiting 0 ms 2023-04-27 01:54:18,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-27 01:54:18,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:18,034 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: da5b8d348b0d50311fe8eae652165c86 to self. 2023-04-27 01:54:18,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,036 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=CLOSED 2023-04-27 01:54:18,037 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560458036"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560458036"}]},"ts":"1682560458036"} 2023-04-27 01:54:18,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-27 01:54:18,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 in 161 msec 2023-04-27 01:54:18,039 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=true 2023-04-27 01:54:18,190 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,190 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560458190"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560458190"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560458190"}]},"ts":"1682560458190"} 2023-04-27 01:54:18,192 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:54:18,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => da5b8d348b0d50311fe8eae652165c86, NAME => 'TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:54:18,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:54:18,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,351 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,351 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A 2023-04-27 01:54:18,351 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A 2023-04-27 01:54:18,352 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:18,354 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da5b8d348b0d50311fe8eae652165c86 columnFamilyName A 2023-04-27 01:54:18,356 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(310): Store=da5b8d348b0d50311fe8eae652165c86/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:18,357 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,358 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B 2023-04-27 01:54:18,358 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B 2023-04-27 01:54:18,358 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:18,358 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da5b8d348b0d50311fe8eae652165c86 columnFamilyName B 2023-04-27 01:54:18,359 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(310): Store=da5b8d348b0d50311fe8eae652165c86/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:18,359 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,360 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C 2023-04-27 01:54:18,360 DEBUG [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C 2023-04-27 01:54:18,360 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:18,361 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region da5b8d348b0d50311fe8eae652165c86 columnFamilyName C 2023-04-27 01:54:18,361 INFO [StoreOpener-da5b8d348b0d50311fe8eae652165c86-1] regionserver.HStore(310): Store=da5b8d348b0d50311fe8eae652165c86/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:18,362 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:54:18,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened da5b8d348b0d50311fe8eae652165c86; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70910657, jitterRate=0.05665113031864166}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:54:18,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:18,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., pid=23, masterSystemTime=1682560458344 2023-04-27 01:54:18,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,370 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,370 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560458370"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560458370"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560458370"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560458370"}]},"ts":"1682560458370"} 2023-04-27 01:54:18,373 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-27 01:54:18,373 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 in 180 msec 2023-04-27 01:54:18,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-27 01:54:18,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, REOPEN/MOVE in 499 msec 2023-04-27 01:54:18,377 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-27 01:54:18,377 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 515 msec 2023-04-27 01:54:18,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-04-27 01:54:18,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-27 01:54:18,391 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x05609e0c to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,396 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@542e33ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,397 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x31cba94a to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,401 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695af552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,401 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x7adf5e8e to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,405 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16323966, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,405 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x52f29016 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,413 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@550c7a27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,413 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x06f13c14 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,417 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40c57efc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,417 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5a81b5 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,421 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4183389c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,422 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x45e59be5 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,425 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bccfdc5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,425 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x2668d41d to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,433 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fbb4986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,433 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x6b416c44 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:18,437 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b69850d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:18,451 DEBUG [hconnection-0x61688c4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,451 DEBUG [hconnection-0x4f1c6982-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,452 DEBUG [hconnection-0x53b42b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:18,456 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,457 DEBUG [hconnection-0x7a22c5c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,459 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,465 DEBUG [hconnection-0x5c547a65-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,465 DEBUG [hconnection-0x44a2174c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,465 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,465 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,466 DEBUG [hconnection-0x28e306bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,466 DEBUG [hconnection-0x5dcd6e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,466 DEBUG [hconnection-0x25fd580c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:18,470 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,470 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,479 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:18,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:18,485 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:18,485 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,485 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:18,486 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:18,486 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:18,486 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45148, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:18,487 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,487 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,488 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:18,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:18,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,489 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,489 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:18,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:18,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:18,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:54:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:18,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:18,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:18,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:18,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:18,500 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,500 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:18,500 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:18,502 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:18,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:18,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:18,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:18,504 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:18,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:18,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7611417a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:18,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:18,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,510 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,510 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,510 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,510 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:18,511 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:18,511 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,511 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:18,512 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,512 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:18,512 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:18,512 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:18,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:18,517 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:18,517 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:18,517 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:18,517 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:18,517 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:18,519 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,519 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:18,519 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:18,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:18,519 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,521 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:18,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:18,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:18,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:18,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:18,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:18,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:18,524 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:18,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:18,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:18,523 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:18,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,524 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:18,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560458521, max timestamp=1682560458526Num uniques 859; 2023-04-27 01:54:18,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:18,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560458521, max timestamp=1682560458526Num uniques 859; 2023-04-27 01:54:18,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:18,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560458521, max timestamp=1682560458526Num uniques 859; 2023-04-27 01:54:18,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:18,546 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042773517b9d485f485e8ad58e4dbe833650 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042773517b9d485f485e8ad58e4dbe833650 2023-04-27 01:54:18,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560518544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560518545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560518546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560518549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560518550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,578 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/6b5fec2613304eb3a8b3525177ea4eb1 2023-04-27 01:54:18,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/68f5b9c62566414ea7c729ece43a77f0 2023-04-27 01:54:18,624 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:18,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:18,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:18,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:18,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:18,637 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:18,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:18,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:18,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:18,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,640 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:18,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:18,640 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,641 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,641 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:18,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:18,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560518652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560518652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560518652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560518652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560518653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:18,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:18,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:18,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:18,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:18,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/3e584efcf192401abe73f253a48b792e 2023-04-27 01:54:18,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/6b5fec2613304eb3a8b3525177ea4eb1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/6b5fec2613304eb3a8b3525177ea4eb1 2023-04-27 01:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:18,682 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,682 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:18,682 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:18,682 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:18,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/6b5fec2613304eb3a8b3525177ea4eb1, entries=150, sequenceid=15, filesize=25.1 K 2023-04-27 01:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:18,683 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:18,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/68f5b9c62566414ea7c729ece43a77f0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/68f5b9c62566414ea7c729ece43a77f0 2023-04-27 01:54:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ae3d09e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:18,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:18,689 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/68f5b9c62566414ea7c729ece43a77f0, entries=150, sequenceid=15, filesize=11.6 K 2023-04-27 01:54:18,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:18,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:18,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/3e584efcf192401abe73f253a48b792e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/3e584efcf192401abe73f253a48b792e 2023-04-27 01:54:18,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:18,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:18,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:18,695 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:18,696 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:18,696 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:18,696 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:18,696 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:18,698 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:18,699 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:18,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:18,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:18,703 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:18,703 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:18,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:18,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:18,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:18,703 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:18,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:18,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,704 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:18,704 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:18,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/3e584efcf192401abe73f253a48b792e, entries=150, sequenceid=15, filesize=11.6 K 2023-04-27 01:54:18,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for da5b8d348b0d50311fe8eae652165c86 in 214ms, sequenceid=15, compaction requested=false 2023-04-27 01:54:18,711 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-27 01:54:18,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:18,804 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:18,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:18,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:18,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:18,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:18,816 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:18,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:18,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:18,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:18,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:18,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:18,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:18,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:18,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:18,825 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:18,825 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:18,825 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-27 01:54:18,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:18,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:18,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:18,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:18,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:18,826 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:18,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:18,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:18,863 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042749421f0751e448d78ecde1aa8d7205b2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042749421f0751e448d78ecde1aa8d7205b2 2023-04-27 01:54:18,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560518865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560518864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560518867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560518869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560518869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,888 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/400ffa8151274d259ea5b0fe21a031e5 2023-04-27 01:54:18,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560518970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560518971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560518972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560518974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:18,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:18,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560518975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560519174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560519174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560519175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560519177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560519180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,318 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3 2023-04-27 01:54:19,347 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/8b39e429463540d1a89c153204504aac 2023-04-27 01:54:19,354 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/400ffa8151274d259ea5b0fe21a031e5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/400ffa8151274d259ea5b0fe21a031e5 2023-04-27 01:54:19,360 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/400ffa8151274d259ea5b0fe21a031e5, entries=150, sequenceid=50, filesize=25.1 K 2023-04-27 01:54:19,361 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3 2023-04-27 01:54:19,367 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3, entries=150, sequenceid=50, filesize=11.6 K 2023-04-27 01:54:19,369 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/8b39e429463540d1a89c153204504aac as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8b39e429463540d1a89c153204504aac 2023-04-27 01:54:19,375 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8b39e429463540d1a89c153204504aac, entries=150, sequenceid=50, filesize=11.6 K 2023-04-27 01:54:19,376 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for da5b8d348b0d50311fe8eae652165c86 in 551ms, sequenceid=50, compaction requested=false 2023-04-27 01:54:19,381 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:19,381 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:19,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:19,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:19,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:19,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7c8e5d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:19,387 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:19,387 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,390 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:19,391 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,391 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:19,393 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:19,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:19,397 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:19,397 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:19,397 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:19,397 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:19,397 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:19,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,400 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,401 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:19,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,411 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:19,411 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:19,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:19,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:19,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:19,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:19,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:19,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:19,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:19,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:19,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:19,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560519503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560519505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,511 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:19,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560519506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560519507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560519508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,513 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272d42d0d6d2904814a19e2a8b3becd2de to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d42d0d6d2904814a19e2a8b3becd2de 2023-04-27 01:54:19,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:19,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:19,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:19,544 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:19,544 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,544 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:19,544 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:19,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,547 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,547 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,547 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:19,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:19,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:19,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:19,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:19,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:19,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c46e61b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:19,571 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:19,571 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:19,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:19,576 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:19,581 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:19,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:19,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:19,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:19,581 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:19,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,584 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,588 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,594 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,594 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,595 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,595 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,595 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,595 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:19,595 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:19,595 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,595 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,596 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,596 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560519609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560519609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560519614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560519614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560519615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,695 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:19,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:19,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:19,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:19,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:19,700 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:19,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:19,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:19,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,703 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:19,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:19,705 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,705 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:19,705 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:19,706 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:19,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10206078[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:19,709 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:19,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:19,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:19,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,712 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,712 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,712 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:19,715 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:19,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:19,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:19,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:19,715 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:19,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,718 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,720 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,722 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:19,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,723 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:19,723 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:19,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560519815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560519815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560519819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560519819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:19,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560519820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,823 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:19,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:19,827 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:19,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:19,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:19,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,829 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:19,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:19,830 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,832 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742325c2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:19,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:19,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:19,836 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:19,836 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,836 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:19,837 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:19,840 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:19,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:19,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:19,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:19,840 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:19,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,843 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,845 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,845 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:19,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:19,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,846 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:19,846 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:19,939 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/625228f6b18e4e03b9d825b748febe62 2023-04-27 01:54:19,947 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:19,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:19,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:19,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:19,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:19,953 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:19,953 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,953 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:19,953 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:19,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,958 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:19,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:19,963 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,963 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:19,963 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:19,964 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,965 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e97437d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:19,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:19,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,968 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:19,969 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,969 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:19,971 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:19,974 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:19,974 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:19,974 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:19,974 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:19,974 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:19,976 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,976 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:19,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,976 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:19,976 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,978 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,980 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,980 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,980 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:19,980 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:19,980 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:19,980 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:19,980 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:19,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:19,980 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,981 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:19,981 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:19,981 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:19,981 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,081 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:20,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:20,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:20,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:20,086 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:20,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:20,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:20,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,088 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:20,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:20,099 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,099 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:20,099 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:20,099 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,103 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ba128a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:20,105 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:20,105 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,106 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 01:54:20,108 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:20,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:20,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:20,113 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,113 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:20,114 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560520120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560520120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:20,124 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:20,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:20,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:20,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:20,124 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:20,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560520125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560520125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560520125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,127 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,127 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,127 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,133 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:20,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:20,134 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:20,134 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:20,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,234 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:20,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:20,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:20,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:20,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:20,238 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:20,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:20,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:20,239 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,239 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,240 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,240 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,241 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,241 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:20,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:20,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:20,244 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:20,246 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a83e5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:20,248 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:20,248 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,249 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:20,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:20,251 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:20,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:20,265 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:20,265 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:20,265 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:20,265 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:20,265 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:20,267 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,267 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,267 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:20,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:20,272 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,272 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,272 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,272 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:20,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:20,272 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:20,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/5f103f320c0949c6bae66091312fe297 2023-04-27 01:54:20,372 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:20,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:20,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:20,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:20,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:20,377 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:20,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:20,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:20,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,378 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,379 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,379 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:20,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,381 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:20,394 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6beaabc3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:20,396 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:20,396 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,397 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:20,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:20,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:20,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:20,402 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:20,402 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:20,402 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:20,402 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:20,402 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:20,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,405 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:20,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,409 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,409 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:20,409 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,509 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:20,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:20,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:20,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:20,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:20,514 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:20,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:20,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:20,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,516 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:20,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:20,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:20,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:20,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,521 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f129168[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:20,523 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:20,523 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,526 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,526 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,526 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:20,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:20,528 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:20,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:20,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:20,531 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:20,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:20,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:20,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:20,531 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:20,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,551 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,552 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:20,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:20,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,555 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:20,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,555 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:20,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:20,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560520626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560520626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560520628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560520628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:20,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560520631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,656 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:20,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:20,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:20,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:20,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:20,660 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:20,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:20,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:20,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,662 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,663 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,663 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:20,665 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,665 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:20,665 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:20,665 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:20,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,667 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:20,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a79ee42[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:20,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:20,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,671 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:20,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:20,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:20,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:20,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:20,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,672 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:20,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:20,675 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:20,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:20,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:20,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:20,675 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:20,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:20,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:20,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:20,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:20,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:20,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,684 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:20,684 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:20,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:20,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:20,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/e3a7d817eae84ec4bd098cdb6a44429d 2023-04-27 01:54:20,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/625228f6b18e4e03b9d825b748febe62 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/625228f6b18e4e03b9d825b748febe62 2023-04-27 01:54:20,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/625228f6b18e4e03b9d825b748febe62, entries=250, sequenceid=63, filesize=38.7 K 2023-04-27 01:54:20,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/5f103f320c0949c6bae66091312fe297 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/5f103f320c0949c6bae66091312fe297 2023-04-27 01:54:20,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/5f103f320c0949c6bae66091312fe297, entries=150, sequenceid=63, filesize=11.6 K 2023-04-27 01:54:20,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/e3a7d817eae84ec4bd098cdb6a44429d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e3a7d817eae84ec4bd098cdb6a44429d 2023-04-27 01:54:20,785 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:20,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:20,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e3a7d817eae84ec4bd098cdb6a44429d, entries=150, sequenceid=63, filesize=11.6 K 2023-04-27 01:54:20,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:20,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for da5b8d348b0d50311fe8eae652165c86 in 1309ms, sequenceid=63, compaction requested=true 2023-04-27 01:54:20,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:20,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:20,788 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:54:20,788 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:54:20,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:20,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:20,789 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:54:20,790 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/A is initiating minor compaction (all files) 2023-04-27 01:54:20,790 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:54:20,790 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/B is initiating minor compaction (all files) 2023-04-27 01:54:20,790 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/B in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,790 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/68f5b9c62566414ea7c729ece43a77f0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/5f103f320c0949c6bae66091312fe297] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=34.8 K 2023-04-27 01:54:20,790 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/A in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,790 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/6b5fec2613304eb3a8b3525177ea4eb1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/400ffa8151274d259ea5b0fe21a031e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/625228f6b18e4e03b9d825b748febe62] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=89.0 K 2023-04-27 01:54:20,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:20,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:20,791 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 68f5b9c62566414ea7c729ece43a77f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682560458491 2023-04-27 01:54:20,791 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:20,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:20,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:20,791 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 6b5fec2613304eb3a8b3525177ea4eb1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682560458491 2023-04-27 01:54:20,791 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e2d9e8ba8db2440a80a3ce6ccd21e5b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682560458545 2023-04-27 01:54:20,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,791 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 400ffa8151274d259ea5b0fe21a031e5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682560458545 2023-04-27 01:54:20,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,792 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 5f103f320c0949c6bae66091312fe297, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682560458861 2023-04-27 01:54:20,792 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 625228f6b18e4e03b9d825b748febe62, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682560458861 2023-04-27 01:54:20,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:20,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:20,793 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:20,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:20,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:20,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:20,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:20,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:20,795 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:20,796 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:20,805 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#B#compaction#126 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:20,844 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#A#compaction#127 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:20,861 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/c13bac0836b24282a10c26a35e689944 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c13bac0836b24282a10c26a35e689944 2023-04-27 01:54:20,867 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/B of da5b8d348b0d50311fe8eae652165c86 into c13bac0836b24282a10c26a35e689944(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:20,867 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:20,867 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/B, priority=13, startTime=1682560460788; duration=0sec 2023-04-27 01:54:20,867 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:20,867 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:54:20,869 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:54:20,869 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/C is initiating minor compaction (all files) 2023-04-27 01:54:20,869 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/C in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:20,869 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/3e584efcf192401abe73f253a48b792e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8b39e429463540d1a89c153204504aac, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e3a7d817eae84ec4bd098cdb6a44429d] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=34.8 K 2023-04-27 01:54:20,870 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3e584efcf192401abe73f253a48b792e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682560458491 2023-04-27 01:54:20,870 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8b39e429463540d1a89c153204504aac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682560458545 2023-04-27 01:54:20,871 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e3a7d817eae84ec4bd098cdb6a44429d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682560458861 2023-04-27 01:54:20,876 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042731377b199c8e400daef3294838f7a580 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042731377b199c8e400daef3294838f7a580 2023-04-27 01:54:20,890 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#C#compaction#129 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:20,913 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/077793f557034cf3a44b54b6224e7a61 2023-04-27 01:54:21,321 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/db6e0cda7f1c442684913ff48fdb266d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/db6e0cda7f1c442684913ff48fdb266d 2023-04-27 01:54:21,329 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/C of da5b8d348b0d50311fe8eae652165c86 into db6e0cda7f1c442684913ff48fdb266d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:21,329 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:21,329 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/C, priority=13, startTime=1682560460788; duration=0sec 2023-04-27 01:54:21,329 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:21,342 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/38bc8e3c4db64b8d93af35e19ad1cf48 2023-04-27 01:54:21,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:21,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:21,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560521641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560521642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560521642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560521643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560521643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,737 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/e343673e4ed94a60be0698ee132197a5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/e343673e4ed94a60be0698ee132197a5 2023-04-27 01:54:21,743 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/A of da5b8d348b0d50311fe8eae652165c86 into e343673e4ed94a60be0698ee132197a5(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:21,743 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:21,743 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/A, priority=13, startTime=1682560460788; duration=0sec 2023-04-27 01:54:21,743 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560521746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560521746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560521747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560521747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,771 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/48be0368898340a3a77a4defd8ece8fd 2023-04-27 01:54:21,776 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/077793f557034cf3a44b54b6224e7a61 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/077793f557034cf3a44b54b6224e7a61 2023-04-27 01:54:21,781 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/077793f557034cf3a44b54b6224e7a61, entries=150, sequenceid=83, filesize=25.1 K 2023-04-27 01:54:21,783 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/38bc8e3c4db64b8d93af35e19ad1cf48 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/38bc8e3c4db64b8d93af35e19ad1cf48 2023-04-27 01:54:21,787 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/38bc8e3c4db64b8d93af35e19ad1cf48, entries=150, sequenceid=83, filesize=11.6 K 2023-04-27 01:54:21,788 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/48be0368898340a3a77a4defd8ece8fd as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/48be0368898340a3a77a4defd8ece8fd 2023-04-27 01:54:21,792 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/48be0368898340a3a77a4defd8ece8fd, entries=150, sequenceid=83, filesize=11.6 K 2023-04-27 01:54:21,793 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for da5b8d348b0d50311fe8eae652165c86 in 997ms, sequenceid=83, compaction requested=false 2023-04-27 01:54:21,793 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:21,793 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:21,795 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:21,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:21,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83ecb43[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:21,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:21,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,799 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:21,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:21,804 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:21,807 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:21,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:21,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:21,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:21,807 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:21,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:21,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:21,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:21,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:21,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:21,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:21,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:21,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:21,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:21,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:21,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:21,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:21,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:21,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:21,813 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:21,813 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:21,913 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:21,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:21,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:21,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:21,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:21,917 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:21,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:21,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:21,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:21,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:21,919 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:21,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:21,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:21,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:21,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:21,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:21,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:21,921 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:21,921 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:21,921 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:21,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:21,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:21,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:21,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:21,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:21,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:21,939 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e486d4a317714bd2a9f8d38ec9aba784 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e486d4a317714bd2a9f8d38ec9aba784 2023-04-27 01:54:21,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:21,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:21,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560521968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560521969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560521969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:21,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:21,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560521969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560522075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560522075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560522076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560522076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560522278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560522279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560522280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560522280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,345 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/b14317c1e4a645f1a727f2699564599e 2023-04-27 01:54:22,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560522583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560522583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560522583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:22,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560522584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,756 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/be161a8104c9411587529ca30d478d88 2023-04-27 01:54:22,768 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/8845b3d4ea8942d6b8d3c747d8a7b904 2023-04-27 01:54:22,773 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/b14317c1e4a645f1a727f2699564599e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b14317c1e4a645f1a727f2699564599e 2023-04-27 01:54:22,778 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b14317c1e4a645f1a727f2699564599e, entries=150, sequenceid=99, filesize=25.1 K 2023-04-27 01:54:22,779 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/be161a8104c9411587529ca30d478d88 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/be161a8104c9411587529ca30d478d88 2023-04-27 01:54:22,783 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/be161a8104c9411587529ca30d478d88, entries=150, sequenceid=99, filesize=11.6 K 2023-04-27 01:54:22,784 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/8845b3d4ea8942d6b8d3c747d8a7b904 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8845b3d4ea8942d6b8d3c747d8a7b904 2023-04-27 01:54:22,789 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8845b3d4ea8942d6b8d3c747d8a7b904, entries=150, sequenceid=99, filesize=11.6 K 2023-04-27 01:54:22,790 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for da5b8d348b0d50311fe8eae652165c86 in 869ms, sequenceid=99, compaction requested=true 2023-04-27 01:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:22,790 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:22,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:22,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3634cd34[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:22,796 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:22,796 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:22,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:22,799 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:22,805 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:22,805 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:22,805 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:22,805 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:22,805 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:22,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:22,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:22,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:22,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:22,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:22,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:22,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:22,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:22,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:22,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:22,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:22,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,814 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:22,814 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:22,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:22,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:22,914 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:22,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:22,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:22,917 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:22,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:22,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:22,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:22,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:22,919 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:22,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:22,920 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:22,920 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:22,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:22,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:22,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:22,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:22,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:22,921 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:22,921 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:22,921 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:22,921 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:22,921 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:22,921 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:22,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:22,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:22,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:23,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:23,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560523096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560523097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560523097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560523098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560523199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560523200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560523205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560523205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,346 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427d7ace99fdb6743bb9cd7df5f76b52e2e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d7ace99fdb6743bb9cd7df5f76b52e2e 2023-04-27 01:54:23,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560523403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560523405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560523409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560523409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560523664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,666 DEBUG [Thread-883] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:54:23,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560523706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560523710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560523714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:23,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560523714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:23,753 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=119, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/24b6da61c9b44b3ab3c1383b2674ffa1 2023-04-27 01:54:24,165 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/1858cfd62f224cb7aecfd04f0f0cf3a0 2023-04-27 01:54:24,187 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/d598ef38bc824999b6878ed470f7fa63 2023-04-27 01:54:24,192 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/24b6da61c9b44b3ab3c1383b2674ffa1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/24b6da61c9b44b3ab3c1383b2674ffa1 2023-04-27 01:54:24,198 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/24b6da61c9b44b3ab3c1383b2674ffa1, entries=150, sequenceid=119, filesize=25.1 K 2023-04-27 01:54:24,198 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/1858cfd62f224cb7aecfd04f0f0cf3a0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1858cfd62f224cb7aecfd04f0f0cf3a0 2023-04-27 01:54:24,203 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1858cfd62f224cb7aecfd04f0f0cf3a0, entries=150, sequenceid=119, filesize=11.6 K 2023-04-27 01:54:24,204 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/d598ef38bc824999b6878ed470f7fa63 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d598ef38bc824999b6878ed470f7fa63 2023-04-27 01:54:24,208 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d598ef38bc824999b6878ed470f7fa63, entries=150, sequenceid=119, filesize=11.6 K 2023-04-27 01:54:24,209 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for da5b8d348b0d50311fe8eae652165c86 in 1288ms, sequenceid=119, compaction requested=true 2023-04-27 01:54:24,209 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:24,209 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:24,210 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:24,210 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,210 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:24,210 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:24,211 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,211 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,212 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,212 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da6934b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:24,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:24,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:24,214 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:24,214 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:24,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:24,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:24,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:24,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:24,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:24,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:24,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:24,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:24,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:24,218 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:24,222 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:24,222 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:24,222 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:24,222 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:24,222 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:24,223 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,224 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,224 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,228 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,229 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,232 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:24,232 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:24,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:24,233 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,233 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:24,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,233 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:24,233 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560524240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560524240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560524242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560524244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,334 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:24,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:24,337 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:24,337 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,337 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:24,337 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:24,338 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,338 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,339 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,339 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,339 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,339 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:24,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:24,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:24,341 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,341 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:24,341 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:24,341 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:24,344 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:24,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28b87143[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:24,346 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:24,346 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560524345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560524346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560524346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,347 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:24,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:24,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:24,348 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,348 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:24,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560524348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,352 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:24,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:24,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:24,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:24,352 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:24,357 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,357 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,357 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,358 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,359 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,364 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,364 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,364 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,364 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,364 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:24,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:24,364 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,365 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,365 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,365 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,365 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,365 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:24,365 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:24,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,465 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:24,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:24,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:24,469 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:24,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:24,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:24,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,470 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,471 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,471 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:24,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:24,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:24,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:24,474 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,474 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5622d3c4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:24,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:24,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,484 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:24,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:24,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:24,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:24,488 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:24,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:24,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:24,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:24,488 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:24,490 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,490 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,490 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,495 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:24,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:24,497 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,497 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,497 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:24,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,498 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560524548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560524548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560524549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560524553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,598 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:24,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:24,601 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:24,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:24,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:24,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:24,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:24,606 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,606 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:24,606 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:24,606 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:24,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:24,608 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466fe959[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:24,610 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:24,610 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:24,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:24,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:24,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:24,612 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:24,615 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:24,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:24,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:24,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:24,615 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:24,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,619 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,620 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,623 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:24,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:24,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:24,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,623 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:24,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,623 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,624 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,624 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,624 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,658 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042753e44e0f7f874f5cab4a70c9c22b5c6f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042753e44e0f7f874f5cab4a70c9c22b5c6f 2023-04-27 01:54:24,671 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=132, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/8898120a5fef4d89bafeb78425e34b6e 2023-04-27 01:54:24,724 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:24,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:24,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:24,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:24,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:24,727 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:24,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:24,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:24,728 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,728 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,729 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:24,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:24,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:24,733 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,733 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,733 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,733 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fda1614[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:24,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:24,735 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,736 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:24,736 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,736 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:24,738 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:24,741 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:24,741 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:24,741 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:24,741 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:24,741 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:24,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,746 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,746 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,749 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,752 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:24,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,753 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,753 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:24,753 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:24,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:24,853 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:24,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:24,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560524852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:24,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560524851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:24,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560524852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,857 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:24,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:24,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:24,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:24,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560524856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,859 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,859 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,859 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,859 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:24,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:24,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:24,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:24,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:24,863 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,863 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:24,865 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248354c5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:24,865 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:24,866 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,866 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:24,869 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:24,872 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:24,872 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:24,872 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:24,872 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:24,872 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:24,873 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,873 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,873 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,875 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,878 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,878 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:24,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:24,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:24,879 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:24,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,879 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:24,879 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:24,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:24,979 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:24,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:24,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:24,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:24,983 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:24,983 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,983 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:24,983 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:24,983 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,984 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,985 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:24,985 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:24,985 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,985 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:24,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:24,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:24,987 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,987 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:24,987 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:24,987 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:24,994 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,994 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647f0f62[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:24,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:24,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,999 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:24,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:24,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:24,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,000 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:25,003 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:25,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:25,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:25,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:25,003 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:25,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,005 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,006 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:25,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,013 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,013 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,013 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,013 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,013 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:25,013 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:25,013 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/a5b57e94be9540e68e0c546be3f4f9f2 2023-04-27 01:54:25,113 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:25,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:25,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:25,116 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:25,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:25,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:25,117 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,117 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,118 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:25,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:25,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:25,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:25,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:25,121 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,122 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,122 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d631795[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:25,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:25,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,124 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:25,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:25,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:25,127 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:25,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:25,130 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:25,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:25,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:25,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:25,130 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:25,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,132 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:25,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:25,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,138 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,138 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,138 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:25,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,239 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:25,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:25,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:25,242 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:25,242 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,242 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:25,242 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:25,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,245 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,246 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,246 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:25,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:25,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:25,249 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4216d09b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:25,251 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:25,251 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,252 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:25,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:25,254 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:25,256 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:25,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:25,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:25,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:25,257 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:25,259 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,259 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,259 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,261 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,263 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,263 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,264 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:25,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:25,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,264 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:25,264 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:25,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560525358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560525358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560525358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,364 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:25,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560525362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:25,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:25,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:25,368 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:25,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:25,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:25,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,370 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,370 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:25,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:25,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:25,371 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,371 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:25,372 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:25,372 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:25,373 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,373 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,373 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,373 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:25,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8d9160[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:25,375 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:25,375 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,382 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,382 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:25,382 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:25,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:25,386 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:25,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:25,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:25,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:25,386 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:25,388 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,388 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,388 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:25,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,392 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,392 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,392 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:25,392 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:25,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:25,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,492 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:25,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:25,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:25,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:25,496 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:25,496 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,496 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:25,496 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:25,496 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,496 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,499 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,499 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,499 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,499 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:25,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:25,501 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,501 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:25,501 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:25,501 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:25,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:25,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:25,503 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/d360140c026a4b0f8089f2471807d088 2023-04-27 01:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6095858c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:25,505 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:25,505 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,507 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,507 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:25,507 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:25,507 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:25,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:25,509 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,510 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:25,510 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,510 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/8898120a5fef4d89bafeb78425e34b6e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/8898120a5fef4d89bafeb78425e34b6e 2023-04-27 01:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:25,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:25,513 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:25,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:25,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:25,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:25,513 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:25,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/8898120a5fef4d89bafeb78425e34b6e, entries=200, sequenceid=132, filesize=32.1 K 2023-04-27 01:54:25,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:25,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/a5b57e94be9540e68e0c546be3f4f9f2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/a5b57e94be9540e68e0c546be3f4f9f2 2023-04-27 01:54:25,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:25,518 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,520 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:25,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:25,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,522 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:25,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:25,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/a5b57e94be9540e68e0c546be3f4f9f2, entries=150, sequenceid=132, filesize=11.7 K 2023-04-27 01:54:25,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:25,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:25,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:25,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,523 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:25,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:25,523 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:25,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/d360140c026a4b0f8089f2471807d088 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d360140c026a4b0f8089f2471807d088 2023-04-27 01:54:25,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d360140c026a4b0f8089f2471807d088, entries=150, sequenceid=132, filesize=11.7 K 2023-04-27 01:54:25,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for da5b8d348b0d50311fe8eae652165c86 in 1315ms, sequenceid=132, compaction requested=true 2023-04-27 01:54:25,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:25,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:25,530 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:25,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:25,530 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:25,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:25,532 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 135923 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:25,532 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:25,533 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/A is initiating minor compaction (all files) 2023-04-27 01:54:25,533 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/B is initiating minor compaction (all files) 2023-04-27 01:54:25,533 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/A in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,533 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/B in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,533 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/e343673e4ed94a60be0698ee132197a5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/077793f557034cf3a44b54b6224e7a61, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b14317c1e4a645f1a727f2699564599e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/24b6da61c9b44b3ab3c1383b2674ffa1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/8898120a5fef4d89bafeb78425e34b6e] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=132.7 K 2023-04-27 01:54:25,533 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c13bac0836b24282a10c26a35e689944, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/38bc8e3c4db64b8d93af35e19ad1cf48, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/be161a8104c9411587529ca30d478d88, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1858cfd62f224cb7aecfd04f0f0cf3a0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/a5b57e94be9540e68e0c546be3f4f9f2] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=58.2 K 2023-04-27 01:54:25,533 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e343673e4ed94a60be0698ee132197a5, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682560458861 2023-04-27 01:54:25,533 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c13bac0836b24282a10c26a35e689944, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682560458861 2023-04-27 01:54:25,534 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 077793f557034cf3a44b54b6224e7a61, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682560459492 2023-04-27 01:54:25,534 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 38bc8e3c4db64b8d93af35e19ad1cf48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682560459492 2023-04-27 01:54:25,534 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b14317c1e4a645f1a727f2699564599e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682560461636 2023-04-27 01:54:25,534 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting be161a8104c9411587529ca30d478d88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682560461636 2023-04-27 01:54:25,534 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 24b6da61c9b44b3ab3c1383b2674ffa1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682560461964 2023-04-27 01:54:25,534 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1858cfd62f224cb7aecfd04f0f0cf3a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682560461964 2023-04-27 01:54:25,535 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8898120a5fef4d89bafeb78425e34b6e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560463096 2023-04-27 01:54:25,535 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting a5b57e94be9540e68e0c546be3f4f9f2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560463096 2023-04-27 01:54:25,562 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#B#compaction#142 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:25,566 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#A#compaction#141 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:25,623 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:25,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:25,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:25,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:25,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:25,627 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:25,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:25,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:25,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,628 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,628 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:25,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:25,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:25,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:25,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:25,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:25,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:25,630 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,630 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:25,630 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:54:25,630 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:25,631 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:25,631 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:25,631 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:25,631 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:25,631 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:25,985 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/4d5ad3d87cb6486fb701fa42b9f893f4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/4d5ad3d87cb6486fb701fa42b9f893f4 2023-04-27 01:54:25,991 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/B of da5b8d348b0d50311fe8eae652165c86 into 4d5ad3d87cb6486fb701fa42b9f893f4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:25,991 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:25,991 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/B, priority=11, startTime=1682560465530; duration=0sec 2023-04-27 01:54:25,991 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:25,992 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:25,999 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:25,999 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/C is initiating minor compaction (all files) 2023-04-27 01:54:25,999 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/C in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:25,999 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/db6e0cda7f1c442684913ff48fdb266d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/48be0368898340a3a77a4defd8ece8fd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8845b3d4ea8942d6b8d3c747d8a7b904, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d598ef38bc824999b6878ed470f7fa63, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d360140c026a4b0f8089f2471807d088] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=58.2 K 2023-04-27 01:54:25,999 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting db6e0cda7f1c442684913ff48fdb266d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682560458861 2023-04-27 01:54:26,000 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 48be0368898340a3a77a4defd8ece8fd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682560459492 2023-04-27 01:54:26,001 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8845b3d4ea8942d6b8d3c747d8a7b904, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682560461636 2023-04-27 01:54:26,001 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d598ef38bc824999b6878ed470f7fa63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1682560461964 2023-04-27 01:54:26,002 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d360140c026a4b0f8089f2471807d088, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560463096 2023-04-27 01:54:26,011 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/27f495e572c84ba99a0614c4f19199d5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/27f495e572c84ba99a0614c4f19199d5 2023-04-27 01:54:26,014 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#C#compaction#144 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:26,016 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/A of da5b8d348b0d50311fe8eae652165c86 into 27f495e572c84ba99a0614c4f19199d5(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:26,017 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:26,017 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/A, priority=11, startTime=1682560465530; duration=0sec 2023-04-27 01:54:26,017 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:26,046 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304273faf513314974e76962b89b875ec7a92 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304273faf513314974e76962b89b875ec7a92 2023-04-27 01:54:26,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:26,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:26,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560526372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560526373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560526374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560526376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,424 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/f4e12b7fad5647f2923c88cac15d73ac as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/f4e12b7fad5647f2923c88cac15d73ac 2023-04-27 01:54:26,430 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/C of da5b8d348b0d50311fe8eae652165c86 into f4e12b7fad5647f2923c88cac15d73ac(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:26,430 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:26,430 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/C, priority=11, startTime=1682560465530; duration=0sec 2023-04-27 01:54:26,430 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:26,452 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/f99f5664ab9744bcb49993db5dd1e045 2023-04-27 01:54:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560526477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560526477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560526477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560526477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560526681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560526681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560526682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560526683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,864 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/6042ce553f7940cbbba6b9681eb89539 2023-04-27 01:54:26,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560526984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560526984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560526985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:26,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:26,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560526986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,277 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/9310685781e84fd785d1962b0ce19e0c 2023-04-27 01:54:27,283 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/f99f5664ab9744bcb49993db5dd1e045 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/f99f5664ab9744bcb49993db5dd1e045 2023-04-27 01:54:27,287 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/f99f5664ab9744bcb49993db5dd1e045, entries=150, sequenceid=151, filesize=25.3 K 2023-04-27 01:54:27,288 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/6042ce553f7940cbbba6b9681eb89539 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/6042ce553f7940cbbba6b9681eb89539 2023-04-27 01:54:27,292 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/6042ce553f7940cbbba6b9681eb89539, entries=150, sequenceid=151, filesize=11.8 K 2023-04-27 01:54:27,293 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/9310685781e84fd785d1962b0ce19e0c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9310685781e84fd785d1962b0ce19e0c 2023-04-27 01:54:27,297 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9310685781e84fd785d1962b0ce19e0c, entries=150, sequenceid=151, filesize=11.8 K 2023-04-27 01:54:27,298 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for da5b8d348b0d50311fe8eae652165c86 in 1668ms, sequenceid=151, compaction requested=false 2023-04-27 01:54:27,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:27,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:27,304 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@258fe9b4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:27,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:27,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,307 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:27,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:27,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:27,314 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:27,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:27,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:27,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:27,314 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:27,315 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,315 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,315 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:27,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:27,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:27,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:27,319 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:27,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:27,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:27,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:27,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:27,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:27,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:27,324 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:27,324 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:27,424 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:27,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:27,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:27,428 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:27,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:27,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:27,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:27,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:27,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:27,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:27,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:27,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:27,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:27,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:27,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:27,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:27,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:27,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:27,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:27,432 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:27,432 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:27,432 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:27,432 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:27,433 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:27,473 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042715db6219d5e74635a5ef1f8c842ad895 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042715db6219d5e74635a5ef1f8c842ad895 2023-04-27 01:54:27,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:27,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:27,504 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/2296bf4710a143c884eac7bf178a8ec2 2023-04-27 01:54:27,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560527507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560527510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560527511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560527511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560527612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560527614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560527615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560527615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560527673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,676 DEBUG [Thread-883] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:54:27,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560527817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560527820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560527820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560527821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:27,920 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/78253a2366dc4675a28c6c87538442cc 2023-04-27 01:54:28,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560528121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560528124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560528125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560528126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,340 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/6ad734faf381496aa5a6e9706d5b2ec8 2023-04-27 01:54:28,345 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/2296bf4710a143c884eac7bf178a8ec2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/2296bf4710a143c884eac7bf178a8ec2 2023-04-27 01:54:28,349 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/2296bf4710a143c884eac7bf178a8ec2, entries=150, sequenceid=167, filesize=25.3 K 2023-04-27 01:54:28,351 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/78253a2366dc4675a28c6c87538442cc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/78253a2366dc4675a28c6c87538442cc 2023-04-27 01:54:28,355 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/78253a2366dc4675a28c6c87538442cc, entries=150, sequenceid=167, filesize=11.8 K 2023-04-27 01:54:28,357 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/6ad734faf381496aa5a6e9706d5b2ec8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/6ad734faf381496aa5a6e9706d5b2ec8 2023-04-27 01:54:28,362 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/6ad734faf381496aa5a6e9706d5b2ec8, entries=150, sequenceid=167, filesize=11.8 K 2023-04-27 01:54:28,362 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for da5b8d348b0d50311fe8eae652165c86 in 930ms, sequenceid=167, compaction requested=true 2023-04-27 01:54:28,362 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:28,363 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:28,363 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:28,365 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:28,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51696796[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:28,367 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:28,367 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,368 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:28,368 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:28,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:28,370 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:28,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:28,373 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:28,373 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:28,373 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:28,373 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:28,373 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:28,374 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,374 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:28,374 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,375 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:28,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:28,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:28,376 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,379 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,379 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:28,379 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:28,380 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:28,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:28,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:28,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:28,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:28,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:28,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:28,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:28,380 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:28,380 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:28,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:28,480 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:28,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:28,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:28,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:28,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:28,484 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:28,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:28,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:28,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:28,485 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,485 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:28,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:28,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:28,486 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:28,487 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,487 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:28,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:28,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:28,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:28,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:28,488 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:28,488 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:28,488 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:28,489 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:28,489 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:28,489 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:28,489 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:28,489 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:28,489 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:28,489 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:28,489 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:28,490 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:28,490 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:28,490 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:28,503 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042743200efd0e3f4d698da3f1a4e86456f8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042743200efd0e3f4d698da3f1a4e86456f8 2023-04-27 01:54:28,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:28,627 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:28,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560528632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560528633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560528633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560528634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560528735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560528736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560528736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560528736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,910 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/bcb5c23fe96c4489a10e08ff854ff072 2023-04-27 01:54:28,935 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/17035dacd68148df8ba639502a261eef 2023-04-27 01:54:28,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560528939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560528940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560528940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560528940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,963 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/149b8dc9de774713bed680843e307dee 2023-04-27 01:54:28,969 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/bcb5c23fe96c4489a10e08ff854ff072 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/bcb5c23fe96c4489a10e08ff854ff072 2023-04-27 01:54:28,974 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/bcb5c23fe96c4489a10e08ff854ff072, entries=150, sequenceid=187, filesize=25.3 K 2023-04-27 01:54:28,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/17035dacd68148df8ba639502a261eef as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/17035dacd68148df8ba639502a261eef 2023-04-27 01:54:28,979 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/17035dacd68148df8ba639502a261eef, entries=150, sequenceid=187, filesize=11.8 K 2023-04-27 01:54:28,980 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/149b8dc9de774713bed680843e307dee as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/149b8dc9de774713bed680843e307dee 2023-04-27 01:54:28,989 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/149b8dc9de774713bed680843e307dee, entries=150, sequenceid=187, filesize=11.8 K 2023-04-27 01:54:28,990 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for da5b8d348b0d50311fe8eae652165c86 in 501ms, sequenceid=187, compaction requested=true 2023-04-27 01:54:28,990 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:28,990 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:28,991 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:28,991 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:28,991 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:28,992 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:28,992 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:29,000 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:29,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417fea4a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:29,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:29,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:29,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:29,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:29,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:29,010 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:29,010 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:29,010 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:29,010 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:29,010 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:29,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:29,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:29,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,014 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:29,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:29,016 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:29,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:29,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:29,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:29,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:29,017 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:29,017 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,017 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,017 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:29,017 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,017 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:29,017 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:29,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:29,117 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:29,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:29,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:29,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:29,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:29,121 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:29,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:29,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:29,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:29,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:29,123 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:29,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:29,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:29,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:29,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:29,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:29,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:29,125 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:29,125 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:29,126 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:54:29,126 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:29,126 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:29,126 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:29,126 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:29,126 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:29,126 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:29,146 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304278f191d3595224821a7e30a3ab1bbfb1f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278f191d3595224821a7e30a3ab1bbfb1f 2023-04-27 01:54:29,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:29,254 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:29,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560529289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560529298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560529298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560529299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560529400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560529401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560529401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560529402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,579 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/d7704ed60fcc46aeab1a4b7436c61303 2023-04-27 01:54:29,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560529603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560529603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560529603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560529604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560529904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560529907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560529907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:29,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560529908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:29,992 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/47ad03f2190d45e38fa25964ce15341c 2023-04-27 01:54:30,012 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/e5f6f982fbeb4d10a01db48e4785889d 2023-04-27 01:54:30,017 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/d7704ed60fcc46aeab1a4b7436c61303 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d7704ed60fcc46aeab1a4b7436c61303 2023-04-27 01:54:30,022 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d7704ed60fcc46aeab1a4b7436c61303, entries=100, sequenceid=199, filesize=18.5 K 2023-04-27 01:54:30,024 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/47ad03f2190d45e38fa25964ce15341c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47ad03f2190d45e38fa25964ce15341c 2023-04-27 01:54:30,029 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47ad03f2190d45e38fa25964ce15341c, entries=100, sequenceid=199, filesize=9.4 K 2023-04-27 01:54:30,030 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/e5f6f982fbeb4d10a01db48e4785889d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e5f6f982fbeb4d10a01db48e4785889d 2023-04-27 01:54:30,034 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e5f6f982fbeb4d10a01db48e4785889d, entries=100, sequenceid=199, filesize=9.4 K 2023-04-27 01:54:30,035 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for da5b8d348b0d50311fe8eae652165c86 in 908ms, sequenceid=199, compaction requested=true 2023-04-27 01:54:30,035 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:30,035 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:30,037 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,037 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:30,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d0aae0c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:30,039 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:30,039 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:30,041 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,041 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:30,045 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:30,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:30,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:30,055 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:30,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:30,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:30,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:30,055 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:30,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:30,060 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:30,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:30,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:30,064 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:30,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:30,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,064 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:30,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:30,064 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:30,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:30,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:30,164 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:30,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:30,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:30,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:30,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:30,168 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:30,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:30,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:30,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:30,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:30,169 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,169 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:30,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:30,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:30,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:30,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:30,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:30,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:30,171 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:30,171 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:30,171 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:30,171 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:30,172 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:30,185 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427351f9bf1a597439eb9333ae00d700621 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427351f9bf1a597439eb9333ae00d700621 2023-04-27 01:54:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:30,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:30,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560530419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560530419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560530420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45142 deadline: 1682560530421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1682560530523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560530523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45116 deadline: 1682560530524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560530524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,591 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=219, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/02e51bee95b14c32a91cab0e7f6ac27c 2023-04-27 01:54:30,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560530726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1682560530727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560530727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:30,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:30,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560530727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,002 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/eb72643e33ae4cb19ea20ab100db9251 2023-04-27 01:54:31,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560531030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560531031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560531031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560531033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,413 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/9387d0d92b64482b835b1171f4ece938 2023-04-27 01:54:31,418 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/02e51bee95b14c32a91cab0e7f6ac27c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/02e51bee95b14c32a91cab0e7f6ac27c 2023-04-27 01:54:31,422 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/02e51bee95b14c32a91cab0e7f6ac27c, entries=150, sequenceid=219, filesize=25.3 K 2023-04-27 01:54:31,423 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/eb72643e33ae4cb19ea20ab100db9251 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/eb72643e33ae4cb19ea20ab100db9251 2023-04-27 01:54:31,428 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/eb72643e33ae4cb19ea20ab100db9251, entries=150, sequenceid=219, filesize=11.8 K 2023-04-27 01:54:31,429 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/9387d0d92b64482b835b1171f4ece938 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9387d0d92b64482b835b1171f4ece938 2023-04-27 01:54:31,434 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9387d0d92b64482b835b1171f4ece938, entries=150, sequenceid=219, filesize=11.8 K 2023-04-27 01:54:31,435 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for da5b8d348b0d50311fe8eae652165c86 in 1263ms, sequenceid=219, compaction requested=true 2023-04-27 01:54:31,435 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:31,435 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57122450[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:31,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:31,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:31,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:31,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:31,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:31,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:31,444 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:31,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:31,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:31,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:31,444 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:31,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,448 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:31,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:31,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,450 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,451 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:31,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,451 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:31,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:31,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:31,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:31,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:31,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:31,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:31,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:31,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:31,551 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:31,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:31,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:31,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:31,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:31,554 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:31,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:31,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:31,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,556 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:31,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:31,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:31,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:31,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:31,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:31,559 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:31,560 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560531557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560531557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560531558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560531558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:31,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dddbcf3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:31,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:31,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:31,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:31,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:31,567 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,567 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:31,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:31,569 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:31,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:31,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:31,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:31,569 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:31,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,573 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,575 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,575 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,575 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,575 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:31,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:31,575 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,576 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,576 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,576 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,576 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,576 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,576 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,576 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:31,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560531662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560531663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560531663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560531663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,676 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:31,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:31,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:31,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:31,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:31,679 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:31,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:31,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:31,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,682 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:31,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:31,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:31,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:31,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:31,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:31,688 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73ce22b8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:31,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:31,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:31,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:31,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:31,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:31,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:31,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:31,699 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:31,699 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:31,699 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:31,699 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:31,699 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:31,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,706 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,712 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:31,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:31,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:31,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,712 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:31,712 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:31,813 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:31,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:31,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:31,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:31,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:31,816 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:31,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:31,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:31,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:31,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:31,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:31,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:31,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:31,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:31,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:31,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bec9803[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:31,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:31,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,824 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:31,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:31,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:31,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:31,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:31,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:31,829 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:31,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:31,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:31,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:31,829 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:31,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,831 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,832 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,834 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:31,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:31,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:31,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,835 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:31,835 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:31,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560531866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560531867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560531867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:31,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560531868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,935 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:31,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:31,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:31,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:31,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:31,939 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:31,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:31,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:31,940 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,940 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:31,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:31,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:31,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:31,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:31,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:31,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:31,945 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,945 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d9a098a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:31,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:31,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:31,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:31,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:31,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:31,954 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427112c805f92984dbba0366b43c9c5ca4b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427112c805f92984dbba0366b43c9c5ca4b 2023-04-27 01:54:31,954 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:31,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:31,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:31,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:31,955 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:31,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:31,963 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,963 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,963 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:31,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,963 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:31,965 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:31,967 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,967 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:31,967 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:31,967 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:31,967 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:31,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:31,967 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:31,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:31,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:31,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:31,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:31,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:31,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:31,968 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:31,968 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:31,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:32,068 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:32,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:32,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:32,071 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:32,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:32,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:32,072 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,072 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,073 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,073 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,073 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,073 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:32,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:32,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:32,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:32,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:32,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:32,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:32,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:32,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d001aee[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:32,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:32,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:32,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:32,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:32,081 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:32,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:32,084 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:32,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:32,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:32,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:32,084 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:32,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:32,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:32,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:32,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,093 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,094 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:32,094 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:32,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560532171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560532171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560532171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560532171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,194 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:32,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:32,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:32,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:32,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:32,197 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:32,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:32,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:32,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:32,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:32,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:32,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:32,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:32,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:32,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:32,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fd93a14[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:32,205 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:32,205 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,206 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:32,206 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:32,206 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:32,206 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,206 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:32,211 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:32,211 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:32,211 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:32,211 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:32,211 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:32,212 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,213 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,213 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,214 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,217 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:32,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:32,217 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,217 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:32,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,318 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:32,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:32,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:32,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:32,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:32,321 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:32,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:32,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:32,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:32,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:32,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:32,324 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:32,324 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,324 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:32,325 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,325 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:32,325 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:32,325 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:32,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:32,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ba8f6c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:32,330 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:32,330 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,331 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,331 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:32,331 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,331 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:32,331 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:32,331 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:32,332 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,332 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,333 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,333 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:32,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:32,335 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:32,335 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:32,335 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:32,335 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:32,335 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:32,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,337 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,337 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,338 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,341 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:32,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:32,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:32,342 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:32,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,342 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:32,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,361 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/1a43740da22547618415f4360f03e0e3 2023-04-27 01:54:32,442 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:32,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:32,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:32,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:32,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:32,445 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:32,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:32,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:32,446 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,446 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:32,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:32,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:32,449 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,449 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:32,449 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:32,449 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:32,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:32,451 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,451 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,451 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:32,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd110a8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:32,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:32,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,454 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:32,454 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:32,454 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:32,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:32,456 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,456 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,457 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,457 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:32,459 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:32,459 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:32,459 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:32,459 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:32,459 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:32,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,480 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,480 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,480 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,480 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,480 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:32,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:32,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,480 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,480 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,480 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,481 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,481 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,481 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:32,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:32,481 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:32,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,581 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:32,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:32,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:32,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:32,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:32,584 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:32,584 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:32,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:32,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,587 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,588 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,588 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:32,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:32,589 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,589 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:32,589 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:32,590 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:32,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:32,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:32,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:32,591 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,591 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70cbaa96[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:32,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:32,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,594 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:32,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:32,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:32,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:32,596 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:32,598 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:32,599 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:32,599 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:32,599 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:32,599 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:32,602 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,602 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,602 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:32,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,606 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,606 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:32,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,606 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:32,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,607 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560532673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560532673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560532676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:32,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560532676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,707 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:32,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:32,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:32,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:32,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:32,710 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:32,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:32,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:32,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,712 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,712 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,712 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,712 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:32,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:32,714 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,714 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:32,714 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:32,714 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:32,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,716 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,716 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,716 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c42701[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:32,718 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:32,718 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,724 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:32,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:32,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:32,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:32,726 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:32,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:32,729 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:32,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:32,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:32,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:32,729 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:32,730 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,730 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,730 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:32,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:32,731 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,738 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,740 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:32,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:32,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:32,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:32,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:32,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:32,741 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:32,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:32,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:32,741 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:32,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/31184dfe408840958732e3a37a4b80f6 2023-04-27 01:54:32,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/975adf4cc5dd46069a0f2ecc41db4d6a 2023-04-27 01:54:32,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/1a43740da22547618415f4360f03e0e3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/1a43740da22547618415f4360f03e0e3 2023-04-27 01:54:32,794 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/1a43740da22547618415f4360f03e0e3, entries=200, sequenceid=232, filesize=32.1 K 2023-04-27 01:54:32,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/31184dfe408840958732e3a37a4b80f6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/31184dfe408840958732e3a37a4b80f6 2023-04-27 01:54:32,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/31184dfe408840958732e3a37a4b80f6, entries=150, sequenceid=232, filesize=11.8 K 2023-04-27 01:54:32,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/975adf4cc5dd46069a0f2ecc41db4d6a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/975adf4cc5dd46069a0f2ecc41db4d6a 2023-04-27 01:54:32,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/975adf4cc5dd46069a0f2ecc41db4d6a, entries=150, sequenceid=232, filesize=11.8 K 2023-04-27 01:54:32,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for da5b8d348b0d50311fe8eae652165c86 in 1270ms, sequenceid=232, compaction requested=true 2023-04-27 01:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:32,806 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 01:54:32,806 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 01:54:32,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:32,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:32,809 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 181488 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 01:54:32,809 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/A is initiating minor compaction (all files) 2023-04-27 01:54:32,809 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/A in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,809 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/27f495e572c84ba99a0614c4f19199d5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/f99f5664ab9744bcb49993db5dd1e045, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/2296bf4710a143c884eac7bf178a8ec2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/bcb5c23fe96c4489a10e08ff854ff072, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d7704ed60fcc46aeab1a4b7436c61303, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/02e51bee95b14c32a91cab0e7f6ac27c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/1a43740da22547618415f4360f03e0e3] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=177.2 K 2023-04-27 01:54:32,809 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82082 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 01:54:32,809 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/B is initiating minor compaction (all files) 2023-04-27 01:54:32,810 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/B in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,810 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/4d5ad3d87cb6486fb701fa42b9f893f4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/6042ce553f7940cbbba6b9681eb89539, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/78253a2366dc4675a28c6c87538442cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/17035dacd68148df8ba639502a261eef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47ad03f2190d45e38fa25964ce15341c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/eb72643e33ae4cb19ea20ab100db9251, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/31184dfe408840958732e3a37a4b80f6] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=80.2 K 2023-04-27 01:54:32,810 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 27f495e572c84ba99a0614c4f19199d5, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560463096 2023-04-27 01:54:32,810 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 4d5ad3d87cb6486fb701fa42b9f893f4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560463096 2023-04-27 01:54:32,811 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f99f5664ab9744bcb49993db5dd1e045, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682560464229 2023-04-27 01:54:32,811 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6042ce553f7940cbbba6b9681eb89539, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682560464229 2023-04-27 01:54:32,811 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2296bf4710a143c884eac7bf178a8ec2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682560466367 2023-04-27 01:54:32,821 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting bcb5c23fe96c4489a10e08ff854ff072, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1682560467508 2023-04-27 01:54:32,821 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 78253a2366dc4675a28c6c87538442cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682560466367 2023-04-27 01:54:32,821 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d7704ed60fcc46aeab1a4b7436c61303, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682560468632 2023-04-27 01:54:32,822 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 17035dacd68148df8ba639502a261eef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1682560467508 2023-04-27 01:54:32,822 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 02e51bee95b14c32a91cab0e7f6ac27c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1682560469296 2023-04-27 01:54:32,822 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 1a43740da22547618415f4360f03e0e3, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682560470416 2023-04-27 01:54:32,823 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 47ad03f2190d45e38fa25964ce15341c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682560468632 2023-04-27 01:54:32,824 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting eb72643e33ae4cb19ea20ab100db9251, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1682560469296 2023-04-27 01:54:32,824 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 31184dfe408840958732e3a37a4b80f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682560470418 2023-04-27 01:54:32,841 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:32,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:32,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:32,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:32,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:32,846 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:32,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:32,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:32,846 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#B#compaction#163 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:32,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,847 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,849 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:32,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:32,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:32,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:32,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:32,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:32,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:32,855 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:32,855 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:32,855 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:32,859 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#A#compaction#162 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:32,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:32,860 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:54:32,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:32,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:32,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:32,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:32,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:32,861 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:32,878 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/3a71fe21617c4b1ab6eb9dcf97d7ae32 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/3a71fe21617c4b1ab6eb9dcf97d7ae32 2023-04-27 01:54:32,883 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/B of da5b8d348b0d50311fe8eae652165c86 into 3a71fe21617c4b1ab6eb9dcf97d7ae32(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:32,884 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:32,884 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/B, priority=9, startTime=1682560472806; duration=0sec 2023-04-27 01:54:32,884 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:32,884 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 01:54:32,886 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82082 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 01:54:32,886 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/C is initiating minor compaction (all files) 2023-04-27 01:54:32,886 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/C in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:32,887 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/f4e12b7fad5647f2923c88cac15d73ac, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9310685781e84fd785d1962b0ce19e0c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/6ad734faf381496aa5a6e9706d5b2ec8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/149b8dc9de774713bed680843e307dee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e5f6f982fbeb4d10a01db48e4785889d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9387d0d92b64482b835b1171f4ece938, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/975adf4cc5dd46069a0f2ecc41db4d6a] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=80.2 K 2023-04-27 01:54:32,887 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f4e12b7fad5647f2923c88cac15d73ac, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560463096 2023-04-27 01:54:32,887 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9310685781e84fd785d1962b0ce19e0c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1682560464229 2023-04-27 01:54:32,888 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6ad734faf381496aa5a6e9706d5b2ec8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682560466367 2023-04-27 01:54:32,889 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 149b8dc9de774713bed680843e307dee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1682560467508 2023-04-27 01:54:32,889 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e5f6f982fbeb4d10a01db48e4785889d, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682560468632 2023-04-27 01:54:32,890 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9387d0d92b64482b835b1171f4ece938, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1682560469296 2023-04-27 01:54:32,890 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 975adf4cc5dd46069a0f2ecc41db4d6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682560470418 2023-04-27 01:54:32,909 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042705688889165946f6b33c9ed4d80c0840 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042705688889165946f6b33c9ed4d80c0840 2023-04-27 01:54:32,915 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#C#compaction#165 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:32,927 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/94fe656cc3b84bd1be8f494fc2f2561e 2023-04-27 01:54:33,314 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/54142780d36e47cf88edc39f94b4d374 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/54142780d36e47cf88edc39f94b4d374 2023-04-27 01:54:33,326 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/A of da5b8d348b0d50311fe8eae652165c86 into 54142780d36e47cf88edc39f94b4d374(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:33,326 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:33,326 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/A, priority=9, startTime=1682560472806; duration=0sec 2023-04-27 01:54:33,326 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:33,347 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/5f90854c1c8d441288ecb55d3b5bf8ef as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/5f90854c1c8d441288ecb55d3b5bf8ef 2023-04-27 01:54:33,356 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/b37e2a72b6704dd3a40fcb934876161b 2023-04-27 01:54:33,357 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/C of da5b8d348b0d50311fe8eae652165c86 into 5f90854c1c8d441288ecb55d3b5bf8ef(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:33,357 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:33,357 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/C, priority=9, startTime=1682560472806; duration=0sec 2023-04-27 01:54:33,357 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:33,367 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/b274f1a69ced4a0887a74afd2aa5d78a 2023-04-27 01:54:33,373 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/94fe656cc3b84bd1be8f494fc2f2561e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/94fe656cc3b84bd1be8f494fc2f2561e 2023-04-27 01:54:33,378 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/94fe656cc3b84bd1be8f494fc2f2561e, entries=150, sequenceid=251, filesize=25.3 K 2023-04-27 01:54:33,379 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/b37e2a72b6704dd3a40fcb934876161b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/b37e2a72b6704dd3a40fcb934876161b 2023-04-27 01:54:33,384 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/b37e2a72b6704dd3a40fcb934876161b, entries=150, sequenceid=251, filesize=11.8 K 2023-04-27 01:54:33,385 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/b274f1a69ced4a0887a74afd2aa5d78a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/b274f1a69ced4a0887a74afd2aa5d78a 2023-04-27 01:54:33,394 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/b274f1a69ced4a0887a74afd2aa5d78a, entries=150, sequenceid=251, filesize=11.8 K 2023-04-27 01:54:33,395 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for da5b8d348b0d50311fe8eae652165c86 in 535ms, sequenceid=251, compaction requested=false 2023-04-27 01:54:33,395 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:33,395 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:33,397 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,398 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,398 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2251175a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:33,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:33,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,402 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:33,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:33,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:33,402 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,402 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:33,403 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,403 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:33,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:33,407 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:33,407 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:33,407 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:33,408 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:33,408 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:33,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,411 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,412 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,415 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:33,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:33,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:33,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,416 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,416 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:33,517 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:33,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:33,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:33,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:33,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:33,521 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:33,521 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,521 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:33,521 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:33,523 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,523 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,524 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:33,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:33,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:33,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:33,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:33,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:33,536 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,536 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:33,536 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:33,536 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:33,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:33,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,541 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:33,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b007e8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:33,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:33,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,544 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:33,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:33,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:33,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:33,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:33,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,546 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:33,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:33,549 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:33,549 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:33,549 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:33,549 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:33,549 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:33,552 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,552 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,552 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:33,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:33,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:33,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,558 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:33,558 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:33,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,558 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,558 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,558 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,558 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,658 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:33,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:33,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:33,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:33,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:33,662 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:33,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:33,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:33,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,664 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:33,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:33,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:33,666 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,666 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:33,666 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:33,666 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:33,668 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,669 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:33,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a828092[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:33,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:33,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,672 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:33,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:33,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:33,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:33,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:33,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:33,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:33,677 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:33,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:33,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:33,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:33,678 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:33,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,685 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:33,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,685 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:33,686 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:33,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:33,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:54:33,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:33,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:33,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:33,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:33,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:33,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:33,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560533745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560533749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560533750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682560473733, max timestamp=1682560473750Num uniques 859; 2023-04-27 01:54:33,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682560473733, max timestamp=1682560473750Num uniques 859; 2023-04-27 01:54:33,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 01:54:33,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560473743, max timestamp=1682560473750Num uniques 859; 2023-04-27 01:54:33,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:33,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 01:54:33,786 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:33,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:33,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:33,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:33,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:33,789 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:33,789 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,789 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:33,789 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:33,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,792 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:33,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:33,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:33,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:33,794 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,794 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:33,794 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:33,794 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:33,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:33,796 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:33,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da944ec[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:33,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:33,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,799 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560533797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:33,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:33,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:33,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:33,801 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:33,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:33,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:33,804 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:33,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:33,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:33,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:33,804 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,811 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:33,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:33,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,812 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:33,812 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560533852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560533852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560533852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:33,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560533900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,912 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:33,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:33,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:33,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:33,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:33,915 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:33,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:33,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:33,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:33,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:33,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:33,918 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:33,919 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,919 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:33,919 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:33,919 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:33,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:33,920 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:33,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476865c4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:33,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:33,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,931 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:33,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:33,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:33,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:33,932 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:33,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:33,935 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:33,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:33,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:33,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:33,935 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:33,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:33,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:33,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:33,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:33,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:33,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:33,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:33,942 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:33,942 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:33,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,042 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,045 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,048 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:34,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5070a586[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,055 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560534057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560534057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,059 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,060 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,060 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,060 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,060 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560534058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,064 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,069 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,069 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560534102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,157 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e5b0fc556ba549fbbfe794c29b66bab1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e5b0fc556ba549fbbfe794c29b66bab1 2023-04-27 01:54:34,169 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,173 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,178 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e58290a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,187 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,187 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,191 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:34,193 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,193 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,293 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,296 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:34,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,303 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f18c09[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,307 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,309 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,309 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,319 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,319 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,319 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:34,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560534358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560534360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560534363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560534408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,419 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,422 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,425 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:34,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,427 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db384b6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,431 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,433 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,435 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,435 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,435 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,435 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,435 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,442 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,442 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,442 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:34,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,542 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,545 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,550 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,550 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,550 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,550 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8ba4613[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,554 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,556 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,558 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,558 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,560 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,560 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,560 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,562 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,565 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,565 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,566 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=267, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/d1e18b33e8154d09b31a498b8164374d 2023-04-27 01:54:34,665 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,668 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,669 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,670 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,672 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,672 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,672 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,672 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37152e77[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,682 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,682 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,685 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,694 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,694 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,794 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,799 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,800 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:34,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,803 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,803 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,803 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,804 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,805 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62516d58[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,811 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,812 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,814 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,814 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:34,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,820 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,820 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560534865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560534865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560534868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:34,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560534911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,920 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:34,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:34,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:34,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:34,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:34,924 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:34,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:34,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:34,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,926 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:34,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:34,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:34,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:34,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:34,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:34,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:34,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:34,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:34,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:34,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:34,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:34,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:34,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:34,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,930 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:34,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2306c390[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:34,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:34,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,932 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:34,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:34,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:34,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:34,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:34,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:34,936 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:34,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:34,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:34,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:34,936 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:34,938 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,938 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,938 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:34,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,938 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:34,943 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:34,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:34,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:34,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:34,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:34,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:34,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:34,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:34,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:34,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:34,944 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:34,944 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:34,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:34,977 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/47cdf4104a7e4f1d8d87ea14dabbeaae 2023-04-27 01:54:35,044 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:35,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:35,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:35,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:35,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:35,048 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:35,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:35,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:35,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,049 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:35,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:35,050 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,050 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:35,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:35,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:35,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:35,052 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:35,053 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,054 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:35,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7956833d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:35,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:35,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:35,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,059 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,059 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:35,059 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:35,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:35,061 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:35,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:35,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:35,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:35,061 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:35,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,064 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:35,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:35,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:35,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,070 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:35,070 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:35,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:35,171 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:35,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:35,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:35,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:35,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:35,174 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:35,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:35,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:35,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:35,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:35,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:35,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:35,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:35,178 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,179 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:35,179 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:35,179 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:35,180 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:35,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@672e7cf6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:35,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:35,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:35,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:35,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:35,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:35,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:35,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:35,187 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:35,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:35,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:35,188 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:35,188 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:35,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:35,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:35,191 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:35,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,195 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:35,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:35,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:35,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:35,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:35,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,195 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:35,196 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:35,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:35,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:35,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,296 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:35,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:35,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:35,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:35,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:35,299 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:35,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:35,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:35,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,300 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:35,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:35,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:35,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:35,302 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,302 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:35,302 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:35,302 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:35,304 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:35,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@efce6e6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:35,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:35,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,308 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:35,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:35,309 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:35,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:35,312 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:35,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:35,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:35,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:35,312 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:35,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:35,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:35,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:35,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:35,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:35,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:35,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:35,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:35,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:35,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:35,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:35,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,318 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:35,318 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:35,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:35,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/858da910ba7c4597aca04c49da6aaca4 2023-04-27 01:54:35,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/d1e18b33e8154d09b31a498b8164374d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d1e18b33e8154d09b31a498b8164374d 2023-04-27 01:54:35,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d1e18b33e8154d09b31a498b8164374d, entries=400, sequenceid=267, filesize=59.8 K 2023-04-27 01:54:35,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/47cdf4104a7e4f1d8d87ea14dabbeaae as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47cdf4104a7e4f1d8d87ea14dabbeaae 2023-04-27 01:54:35,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47cdf4104a7e4f1d8d87ea14dabbeaae, entries=100, sequenceid=267, filesize=9.5 K 2023-04-27 01:54:35,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/858da910ba7c4597aca04c49da6aaca4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/858da910ba7c4597aca04c49da6aaca4 2023-04-27 01:54:35,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/858da910ba7c4597aca04c49da6aaca4, entries=100, sequenceid=267, filesize=9.5 K 2023-04-27 01:54:35,410 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for da5b8d348b0d50311fe8eae652165c86 in 1691ms, sequenceid=267, compaction requested=true 2023-04-27 01:54:35,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:35,410 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:35,410 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:35,410 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:54:35,410 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:54:35,410 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:35,412 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34326 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:54:35,412 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/B is initiating minor compaction (all files) 2023-04-27 01:54:35,412 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/B in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,412 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/3a71fe21617c4b1ab6eb9dcf97d7ae32, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/b37e2a72b6704dd3a40fcb934876161b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47cdf4104a7e4f1d8d87ea14dabbeaae] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=33.5 K 2023-04-27 01:54:35,412 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3a71fe21617c4b1ab6eb9dcf97d7ae32, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682560470418 2023-04-27 01:54:35,413 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b37e2a72b6704dd3a40fcb934876161b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682560471550 2023-04-27 01:54:35,414 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 47cdf4104a7e4f1d8d87ea14dabbeaae, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682560473718 2023-04-27 01:54:35,414 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113554 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:54:35,414 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/A is initiating minor compaction (all files) 2023-04-27 01:54:35,414 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/A in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,414 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/54142780d36e47cf88edc39f94b4d374, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/94fe656cc3b84bd1be8f494fc2f2561e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d1e18b33e8154d09b31a498b8164374d] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=110.9 K 2023-04-27 01:54:35,415 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 54142780d36e47cf88edc39f94b4d374, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682560470418 2023-04-27 01:54:35,415 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 94fe656cc3b84bd1be8f494fc2f2561e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682560471550 2023-04-27 01:54:35,416 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d1e18b33e8154d09b31a498b8164374d, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682560473706 2023-04-27 01:54:35,418 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:35,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:35,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:35,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:35,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:35,426 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:35,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:35,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:35,427 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,427 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,427 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#B#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:35,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:35,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:35,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:35,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:35,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:35,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:35,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:35,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:35,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:35,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:35,433 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#A#compaction#172 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:35,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:35,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:35,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:35,440 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=131.94 KB heapSize=361.16 KB 2023-04-27 01:54:35,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:35,441 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:35,441 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:35,441 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:35,441 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:35,441 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:35,473 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/1e2d0e3a06354f08a29ec44d041cac1e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1e2d0e3a06354f08a29ec44d041cac1e 2023-04-27 01:54:35,489 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/B of da5b8d348b0d50311fe8eae652165c86 into 1e2d0e3a06354f08a29ec44d041cac1e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:35,489 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:35,489 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/B, priority=13, startTime=1682560475410; duration=0sec 2023-04-27 01:54:35,489 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:35,489 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:54:35,490 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34326 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:54:35,490 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/C is initiating minor compaction (all files) 2023-04-27 01:54:35,490 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/C in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:35,491 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/5f90854c1c8d441288ecb55d3b5bf8ef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/b274f1a69ced4a0887a74afd2aa5d78a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/858da910ba7c4597aca04c49da6aaca4] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=33.5 K 2023-04-27 01:54:35,491 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 5f90854c1c8d441288ecb55d3b5bf8ef, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682560470418 2023-04-27 01:54:35,492 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b274f1a69ced4a0887a74afd2aa5d78a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1682560471550 2023-04-27 01:54:35,492 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 858da910ba7c4597aca04c49da6aaca4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682560473718 2023-04-27 01:54:35,504 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#C#compaction#174 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:54:35,519 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/a47631ec47a7404c9ef58097eb6e0d43 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/a47631ec47a7404c9ef58097eb6e0d43 2023-04-27 01:54:35,526 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/C of da5b8d348b0d50311fe8eae652165c86 into a47631ec47a7404c9ef58097eb6e0d43(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:35,526 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:35,526 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/C, priority=13, startTime=1682560475410; duration=0sec 2023-04-27 01:54:35,526 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:35,879 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:35,879 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042786b96ee7755a4edbb7bbefc7107453f2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042786b96ee7755a4edbb7bbefc7107453f2 2023-04-27 01:54:35,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:35,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560535893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560535894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560535894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:35,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560535920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:35,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560535995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:35,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560535995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:35,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:35,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560535995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:36,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:36,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560536199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560536199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:36,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560536199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,289 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=300, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/cd88293cec144e03b2a1dbf1114259ff 2023-04-27 01:54:36,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:36,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560536502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560536503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560536504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,672 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/08899215c2c44ea787497ae6c805aa25 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/08899215c2c44ea787497ae6c805aa25 2023-04-27 01:54:36,678 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/A of da5b8d348b0d50311fe8eae652165c86 into 08899215c2c44ea787497ae6c805aa25(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 01:54:36,678 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:36,678 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/A, priority=13, startTime=1682560475410; duration=1sec 2023-04-27 01:54:36,679 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:36,703 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/0fca8c811cfa49ed9114ee2e3505065a 2023-04-27 01:54:36,715 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/c26dfbf8b7814a7e9d89b422a356546b 2023-04-27 01:54:36,720 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/cd88293cec144e03b2a1dbf1114259ff as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/cd88293cec144e03b2a1dbf1114259ff 2023-04-27 01:54:36,726 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/cd88293cec144e03b2a1dbf1114259ff, entries=150, sequenceid=300, filesize=25.4 K 2023-04-27 01:54:36,729 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/0fca8c811cfa49ed9114ee2e3505065a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/0fca8c811cfa49ed9114ee2e3505065a 2023-04-27 01:54:36,738 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/0fca8c811cfa49ed9114ee2e3505065a, entries=150, sequenceid=300, filesize=11.9 K 2023-04-27 01:54:36,739 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/c26dfbf8b7814a7e9d89b422a356546b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/c26dfbf8b7814a7e9d89b422a356546b 2023-04-27 01:54:36,747 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/c26dfbf8b7814a7e9d89b422a356546b, entries=150, sequenceid=300, filesize=11.9 K 2023-04-27 01:54:36,748 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~361.11 KB/369776, currentSize=60.38 KB/61830 for da5b8d348b0d50311fe8eae652165c86 in 1308ms, sequenceid=300, compaction requested=false 2023-04-27 01:54:36,748 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:36,748 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:36,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:36,750 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:36,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b73aeea[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:36,753 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:36,753 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,755 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,755 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:36,755 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,755 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:36,755 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:36,755 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:36,757 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:36,758 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:36,758 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:36,758 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:36,759 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:36,759 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:36,760 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:36,760 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:36,760 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:36,762 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,762 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:36,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,762 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:36,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:36,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,763 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:36,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:36,766 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:36,766 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:36,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:36,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:36,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:36,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:36,766 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:36,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:36,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:36,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:36,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,767 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:36,767 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:36,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:36,867 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:36,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:36,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:36,871 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:36,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:36,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:36,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:36,872 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,872 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:36,873 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:36,873 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:36,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:36,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:36,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:36,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:36,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:36,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:36,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:36,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:36,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:36,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:36,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:36,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:36,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:36,875 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:54:36,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:36,876 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:36,876 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:36,876 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:36,876 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:36,876 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:36,895 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427dca9a67b711941d597c01aa63a558aa8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427dca9a67b711941d597c01aa63a558aa8 2023-04-27 01:54:36,901 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/98ccfc3619c94c91b49653d088a398c8 2023-04-27 01:54:37,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:37,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:37,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560537027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560477020, max timestamp=1682560477028Num uniques 859; 2023-04-27 01:54:37,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560477020, max timestamp=1682560477028Num uniques 859; 2023-04-27 01:54:37,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560537029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560477020, max timestamp=1682560477028Num uniques 859; 2023-04-27 01:54:37,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:37,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:37,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:37,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560537074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560537131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560537131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560537176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,313 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/267247875ddc47ee8bce90854b8e1ba3 2023-04-27 01:54:37,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560537335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560537336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560537381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560537638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560537638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560537683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45148 deadline: 1682560537711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,714 DEBUG [Thread-883] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18206 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:54:37,725 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/4d8139c61deb424887e4838c589fee5f 2023-04-27 01:54:37,730 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/98ccfc3619c94c91b49653d088a398c8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/98ccfc3619c94c91b49653d088a398c8 2023-04-27 01:54:37,734 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/98ccfc3619c94c91b49653d088a398c8, entries=150, sequenceid=315, filesize=25.4 K 2023-04-27 01:54:37,735 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/267247875ddc47ee8bce90854b8e1ba3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/267247875ddc47ee8bce90854b8e1ba3 2023-04-27 01:54:37,739 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/267247875ddc47ee8bce90854b8e1ba3, entries=150, sequenceid=315, filesize=11.9 K 2023-04-27 01:54:37,739 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/4d8139c61deb424887e4838c589fee5f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/4d8139c61deb424887e4838c589fee5f 2023-04-27 01:54:37,743 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/4d8139c61deb424887e4838c589fee5f, entries=150, sequenceid=315, filesize=11.9 K 2023-04-27 01:54:37,744 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for da5b8d348b0d50311fe8eae652165c86 in 869ms, sequenceid=315, compaction requested=true 2023-04-27 01:54:37,744 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:37,744 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:37,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:37,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:37,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:37,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:37,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f189d2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:37,751 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:37,751 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,752 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:37,752 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,752 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:37,753 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:37,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:37,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:37,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:37,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:37,756 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:37,756 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:37,756 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:37,756 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:37,756 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:37,757 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,757 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:37,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,757 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:37,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:37,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:37,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:37,768 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:37,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:37,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:37,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:37,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:37,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:37,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:37,768 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:37,768 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:37,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:37,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:37,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:37,769 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:37,769 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,869 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:37,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:37,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:37,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:37,874 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:37,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:37,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:37,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:37,875 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,875 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:37,876 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:37,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:37,876 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:37,877 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:37,877 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:37,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:37,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:37,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:37,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:37,879 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:37,879 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:37,879 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:37,879 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:37,879 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:37,879 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:37,879 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. started... 2023-04-27 01:54:37,879 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-27 01:54:37,879 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:37,879 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:37,880 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:37,880 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:37,880 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:37,880 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:37,912 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042728772339704049ca803f5d3671644480 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042728772339704049ca803f5d3671644480 2023-04-27 01:54:37,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:37,932 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. as already flushing 2023-04-27 01:54:37,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:37,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560537956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:38,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560538060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:38,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45142 deadline: 1682560538144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:38,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45116 deadline: 1682560538145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1682560538190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:38,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560538263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,330 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=349, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/621d3ab9f26047a18add962bc503ac89 2023-04-27 01:54:38,347 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/c7b8a14981074b8f8673d701662049a3 2023-04-27 01:54:38,450 DEBUG [Thread-892] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b416c44 to 127.0.0.1:52731 2023-04-27 01:54:38,450 DEBUG [Thread-892] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:38,451 DEBUG [Thread-886] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5a81b5 to 127.0.0.1:52731 2023-04-27 01:54:38,451 DEBUG [Thread-886] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:38,451 DEBUG [Thread-888] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45e59be5 to 127.0.0.1:52731 2023-04-27 01:54:38,451 DEBUG [Thread-888] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:38,453 DEBUG [Thread-890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2668d41d to 127.0.0.1:52731 2023-04-27 01:54:38,453 DEBUG [Thread-890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:38,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:38,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1682560538567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,762 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/543be770996745a4ad12746ca7c66dee 2023-04-27 01:54:38,766 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/621d3ab9f26047a18add962bc503ac89 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/621d3ab9f26047a18add962bc503ac89 2023-04-27 01:54:38,771 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/621d3ab9f26047a18add962bc503ac89, entries=150, sequenceid=349, filesize=25.4 K 2023-04-27 01:54:38,772 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/c7b8a14981074b8f8673d701662049a3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c7b8a14981074b8f8673d701662049a3 2023-04-27 01:54:38,776 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c7b8a14981074b8f8673d701662049a3, entries=150, sequenceid=349, filesize=11.9 K 2023-04-27 01:54:38,777 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/543be770996745a4ad12746ca7c66dee as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/543be770996745a4ad12746ca7c66dee 2023-04-27 01:54:38,781 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/543be770996745a4ad12746ca7c66dee, entries=150, sequenceid=349, filesize=11.9 K 2023-04-27 01:54:38,781 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for da5b8d348b0d50311fe8eae652165c86 in 902ms, sequenceid=349, compaction requested=true 2023-04-27 01:54:38,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:38,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:38,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:38,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:38,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:38,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:38,783 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,783 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:38,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@422c9636[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:38,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:38,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:38,786 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,786 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:38,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:38,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:38,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:38,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:38,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:38,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:38,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:38,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:38,792 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:38,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:38,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:38,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:38,792 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:38,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:38,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:38,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:38,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:38,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:38,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:38,795 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,796 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:38,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:38,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:38,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:38,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:38,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:38,797 DEBUG [Thread-885] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:38,797 DEBUG [Thread-885] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:38,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:38,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:38,897 DEBUG [Thread-885] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:38,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:39,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:39,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:39,073 DEBUG [Thread-875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05609e0c to 127.0.0.1:52731 2023-04-27 01:54:39,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:39,073 DEBUG [Thread-875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:39,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:39,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:39,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:39,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:39,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:39,149 DEBUG [Thread-879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7adf5e8e to 127.0.0.1:52731 2023-04-27 01:54:39,149 DEBUG [Thread-879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:39,153 DEBUG [Thread-877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31cba94a to 127.0.0.1:52731 2023-04-27 01:54:39,153 DEBUG [Thread-877] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:39,199 DEBUG [Thread-881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52f29016 to 127.0.0.1:52731 2023-04-27 01:54:39,199 DEBUG [Thread-881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:39,485 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e70a7c62e07e432f9e5f8b23d35794a1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e70a7c62e07e432f9e5f8b23d35794a1 2023-04-27 01:54:39,697 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.07692307692307693, tune throughput to 53.85 MB/second 2023-04-27 01:54:39,890 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/b770149f4a254956bb3b8fa0a7d4df04 2023-04-27 01:54:40,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/23b36bd738864418bfc3151b184fef54 2023-04-27 01:54:40,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/245abe3e9fec4ba993ec07d82e3456b3 2023-04-27 01:54:40,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/b770149f4a254956bb3b8fa0a7d4df04 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b770149f4a254956bb3b8fa0a7d4df04 2023-04-27 01:54:40,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b770149f4a254956bb3b8fa0a7d4df04, entries=100, sequenceid=362, filesize=18.6 K 2023-04-27 01:54:40,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/23b36bd738864418bfc3151b184fef54 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/23b36bd738864418bfc3151b184fef54 2023-04-27 01:54:40,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/23b36bd738864418bfc3151b184fef54, entries=100, sequenceid=362, filesize=9.5 K 2023-04-27 01:54:40,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/245abe3e9fec4ba993ec07d82e3456b3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/245abe3e9fec4ba993ec07d82e3456b3 2023-04-27 01:54:40,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/245abe3e9fec4ba993ec07d82e3456b3, entries=100, sequenceid=362, filesize=9.5 K 2023-04-27 01:54:40,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for da5b8d348b0d50311fe8eae652165c86 in 1665ms, sequenceid=362, compaction requested=true 2023-04-27 01:54:40,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:40,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:40,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:40,738 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:40,738 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:40,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:40,739 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 123746 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:40,739 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59050 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:40,739 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/A is initiating minor compaction (all files) 2023-04-27 01:54:40,740 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/B is initiating minor compaction (all files) 2023-04-27 01:54:40,740 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/A in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:40,740 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/B in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:40,740 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/08899215c2c44ea787497ae6c805aa25, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/cd88293cec144e03b2a1dbf1114259ff, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/98ccfc3619c94c91b49653d088a398c8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/621d3ab9f26047a18add962bc503ac89, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b770149f4a254956bb3b8fa0a7d4df04] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=120.8 K 2023-04-27 01:54:40,740 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1e2d0e3a06354f08a29ec44d041cac1e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/0fca8c811cfa49ed9114ee2e3505065a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/267247875ddc47ee8bce90854b8e1ba3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c7b8a14981074b8f8673d701662049a3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/23b36bd738864418bfc3151b184fef54] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=57.7 K 2023-04-27 01:54:40,740 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 08899215c2c44ea787497ae6c805aa25, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682560471550 2023-04-27 01:54:40,740 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1e2d0e3a06354f08a29ec44d041cac1e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682560471550 2023-04-27 01:54:40,740 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cd88293cec144e03b2a1dbf1114259ff, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682560473779 2023-04-27 01:54:40,741 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 0fca8c811cfa49ed9114ee2e3505065a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682560473779 2023-04-27 01:54:40,741 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 98ccfc3619c94c91b49653d088a398c8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682560475886 2023-04-27 01:54:40,741 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 267247875ddc47ee8bce90854b8e1ba3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682560475886 2023-04-27 01:54:40,741 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 621d3ab9f26047a18add962bc503ac89, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682560477047 2023-04-27 01:54:40,741 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c7b8a14981074b8f8673d701662049a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682560477047 2023-04-27 01:54:40,741 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b770149f4a254956bb3b8fa0a7d4df04, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560477949 2023-04-27 01:54:40,742 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 23b36bd738864418bfc3151b184fef54, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560477949 2023-04-27 01:54:40,755 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#B#compaction#187 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:40,762 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#A#compaction#186 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:40,766 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/ac6fc73b88c04cd3b129bb18c5fc759a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/ac6fc73b88c04cd3b129bb18c5fc759a 2023-04-27 01:54:40,771 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/B of da5b8d348b0d50311fe8eae652165c86 into ac6fc73b88c04cd3b129bb18c5fc759a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:40,771 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:40,771 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/B, priority=11, startTime=1682560480738; duration=0sec 2023-04-27 01:54:40,771 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:40,771 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:54:40,772 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59050 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:54:40,772 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): da5b8d348b0d50311fe8eae652165c86/C is initiating minor compaction (all files) 2023-04-27 01:54:40,773 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of da5b8d348b0d50311fe8eae652165c86/C in TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:40,773 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/a47631ec47a7404c9ef58097eb6e0d43, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/c26dfbf8b7814a7e9d89b422a356546b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/4d8139c61deb424887e4838c589fee5f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/543be770996745a4ad12746ca7c66dee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/245abe3e9fec4ba993ec07d82e3456b3] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp, totalSize=57.7 K 2023-04-27 01:54:40,773 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting a47631ec47a7404c9ef58097eb6e0d43, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682560471550 2023-04-27 01:54:40,773 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c26dfbf8b7814a7e9d89b422a356546b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682560473779 2023-04-27 01:54:40,774 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 4d8139c61deb424887e4838c589fee5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682560475886 2023-04-27 01:54:40,774 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 543be770996745a4ad12746ca7c66dee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1682560477047 2023-04-27 01:54:40,774 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 245abe3e9fec4ba993ec07d82e3456b3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560477949 2023-04-27 01:54:40,782 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): da5b8d348b0d50311fe8eae652165c86#C#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:40,791 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/ac2df35c74c34b0ebe09949f58a23e89 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/ac2df35c74c34b0ebe09949f58a23e89 2023-04-27 01:54:40,796 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/C of da5b8d348b0d50311fe8eae652165c86 into ac2df35c74c34b0ebe09949f58a23e89(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:40,796 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:40,796 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/C, priority=11, startTime=1682560480738; duration=0sec 2023-04-27 01:54:40,796 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:41,982 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/9c8c2dc9f44f490ea22ff146aa1fa9db as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/9c8c2dc9f44f490ea22ff146aa1fa9db 2023-04-27 01:54:41,986 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in da5b8d348b0d50311fe8eae652165c86/A of da5b8d348b0d50311fe8eae652165c86 into 9c8c2dc9f44f490ea22ff146aa1fa9db(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 01:54:41,986 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:41,986 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86., storeName=da5b8d348b0d50311fe8eae652165c86/A, priority=11, startTime=1682560480738; duration=1sec 2023-04-27 01:54:41,986 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:47,736 DEBUG [Thread-883] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06f13c14 to 127.0.0.1:52731 2023-04-27 01:54:47,736 DEBUG [Thread-883] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 61 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5238 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5273 2023-04-27 01:54:47,736 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 01:54:47,737 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 2244 2023-04-27 01:54:47,737 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 6732 rows 2023-04-27 01:54:47,737 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 2234 2023-04-27 01:54:47,737 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 6696 rows 2023-04-27 01:54:47,737 INFO [Listener at localhost/42495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 01:54:47,737 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58d22e5a to 127.0.0.1:52731 2023-04-27 01:54:47,737 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:54:47,738 INFO [Listener at localhost/42495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 01:54:47,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 01:54:47,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:47,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 01:54:47,743 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560487743"}]},"ts":"1682560487743"} 2023-04-27 01:54:47,744 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 01:54:47,746 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 01:54:47,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, UNASSIGN}] 2023-04-27 01:54:47,747 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, UNASSIGN 2023-04-27 01:54:47,748 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:47,748 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560487748"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560487748"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560487748"}]},"ts":"1682560487748"} 2023-04-27 01:54:47,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:54:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 01:54:47,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:47,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing da5b8d348b0d50311fe8eae652165c86, disabling compactions & flushes 2023-04-27 01:54:47,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. after waiting 0 ms 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:47,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing da5b8d348b0d50311fe8eae652165c86 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=A 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=B 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK da5b8d348b0d50311fe8eae652165c86, store=C 2023-04-27 01:54:47,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:48,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 01:54:48,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427db614e59af004ee0b9e3356eec4e9d3a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427db614e59af004ee0b9e3356eec4e9d3a 2023-04-27 01:54:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 01:54:48,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/65eb5335f1c64906bebc26753bb3def7 2023-04-27 01:54:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 01:54:49,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/14d89230c8c44744bf39d37e426cfa14 2023-04-27 01:54:49,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/8934075ccc794035a7c590419e2e930d 2023-04-27 01:54:49,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/A/65eb5335f1c64906bebc26753bb3def7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/65eb5335f1c64906bebc26753bb3def7 2023-04-27 01:54:49,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/65eb5335f1c64906bebc26753bb3def7, entries=100, sequenceid=372, filesize=18.6 K 2023-04-27 01:54:49,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/B/14d89230c8c44744bf39d37e426cfa14 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/14d89230c8c44744bf39d37e426cfa14 2023-04-27 01:54:49,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/14d89230c8c44744bf39d37e426cfa14, entries=100, sequenceid=372, filesize=9.5 K 2023-04-27 01:54:49,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/.tmp/C/8934075ccc794035a7c590419e2e930d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8934075ccc794035a7c590419e2e930d 2023-04-27 01:54:49,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8934075ccc794035a7c590419e2e930d, entries=100, sequenceid=372, filesize=9.5 K 2023-04-27 01:54:49,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for da5b8d348b0d50311fe8eae652165c86 in 1656ms, sequenceid=372, compaction requested=false 2023-04-27 01:54:49,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/6b5fec2613304eb3a8b3525177ea4eb1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/400ffa8151274d259ea5b0fe21a031e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/625228f6b18e4e03b9d825b748febe62, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/e343673e4ed94a60be0698ee132197a5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/077793f557034cf3a44b54b6224e7a61, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b14317c1e4a645f1a727f2699564599e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/24b6da61c9b44b3ab3c1383b2674ffa1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/8898120a5fef4d89bafeb78425e34b6e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/27f495e572c84ba99a0614c4f19199d5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/f99f5664ab9744bcb49993db5dd1e045, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/2296bf4710a143c884eac7bf178a8ec2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/bcb5c23fe96c4489a10e08ff854ff072, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d7704ed60fcc46aeab1a4b7436c61303, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/02e51bee95b14c32a91cab0e7f6ac27c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/1a43740da22547618415f4360f03e0e3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/54142780d36e47cf88edc39f94b4d374, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/94fe656cc3b84bd1be8f494fc2f2561e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d1e18b33e8154d09b31a498b8164374d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/08899215c2c44ea787497ae6c805aa25, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/cd88293cec144e03b2a1dbf1114259ff, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/98ccfc3619c94c91b49653d088a398c8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/621d3ab9f26047a18add962bc503ac89, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b770149f4a254956bb3b8fa0a7d4df04] to archive 2023-04-27 01:54:49,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:54:49,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/6b5fec2613304eb3a8b3525177ea4eb1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/6b5fec2613304eb3a8b3525177ea4eb1 2023-04-27 01:54:49,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/400ffa8151274d259ea5b0fe21a031e5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/400ffa8151274d259ea5b0fe21a031e5 2023-04-27 01:54:49,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/625228f6b18e4e03b9d825b748febe62 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/625228f6b18e4e03b9d825b748febe62 2023-04-27 01:54:49,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/e343673e4ed94a60be0698ee132197a5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/e343673e4ed94a60be0698ee132197a5 2023-04-27 01:54:49,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/077793f557034cf3a44b54b6224e7a61 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/077793f557034cf3a44b54b6224e7a61 2023-04-27 01:54:49,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b14317c1e4a645f1a727f2699564599e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b14317c1e4a645f1a727f2699564599e 2023-04-27 01:54:49,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/24b6da61c9b44b3ab3c1383b2674ffa1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/24b6da61c9b44b3ab3c1383b2674ffa1 2023-04-27 01:54:49,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/8898120a5fef4d89bafeb78425e34b6e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/8898120a5fef4d89bafeb78425e34b6e 2023-04-27 01:54:49,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/27f495e572c84ba99a0614c4f19199d5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/27f495e572c84ba99a0614c4f19199d5 2023-04-27 01:54:49,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/f99f5664ab9744bcb49993db5dd1e045 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/f99f5664ab9744bcb49993db5dd1e045 2023-04-27 01:54:49,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/2296bf4710a143c884eac7bf178a8ec2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/2296bf4710a143c884eac7bf178a8ec2 2023-04-27 01:54:49,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/bcb5c23fe96c4489a10e08ff854ff072 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/bcb5c23fe96c4489a10e08ff854ff072 2023-04-27 01:54:49,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d7704ed60fcc46aeab1a4b7436c61303 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d7704ed60fcc46aeab1a4b7436c61303 2023-04-27 01:54:49,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/02e51bee95b14c32a91cab0e7f6ac27c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/02e51bee95b14c32a91cab0e7f6ac27c 2023-04-27 01:54:49,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/1a43740da22547618415f4360f03e0e3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/1a43740da22547618415f4360f03e0e3 2023-04-27 01:54:49,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/54142780d36e47cf88edc39f94b4d374 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/54142780d36e47cf88edc39f94b4d374 2023-04-27 01:54:49,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/94fe656cc3b84bd1be8f494fc2f2561e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/94fe656cc3b84bd1be8f494fc2f2561e 2023-04-27 01:54:49,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d1e18b33e8154d09b31a498b8164374d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/d1e18b33e8154d09b31a498b8164374d 2023-04-27 01:54:49,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/08899215c2c44ea787497ae6c805aa25 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/08899215c2c44ea787497ae6c805aa25 2023-04-27 01:54:49,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/cd88293cec144e03b2a1dbf1114259ff to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/cd88293cec144e03b2a1dbf1114259ff 2023-04-27 01:54:49,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/98ccfc3619c94c91b49653d088a398c8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/98ccfc3619c94c91b49653d088a398c8 2023-04-27 01:54:49,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/621d3ab9f26047a18add962bc503ac89 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/621d3ab9f26047a18add962bc503ac89 2023-04-27 01:54:49,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b770149f4a254956bb3b8fa0a7d4df04 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/b770149f4a254956bb3b8fa0a7d4df04 2023-04-27 01:54:49,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/68f5b9c62566414ea7c729ece43a77f0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c13bac0836b24282a10c26a35e689944, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/5f103f320c0949c6bae66091312fe297, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/38bc8e3c4db64b8d93af35e19ad1cf48, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/be161a8104c9411587529ca30d478d88, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1858cfd62f224cb7aecfd04f0f0cf3a0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/4d5ad3d87cb6486fb701fa42b9f893f4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/a5b57e94be9540e68e0c546be3f4f9f2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/6042ce553f7940cbbba6b9681eb89539, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/78253a2366dc4675a28c6c87538442cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/17035dacd68148df8ba639502a261eef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47ad03f2190d45e38fa25964ce15341c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/eb72643e33ae4cb19ea20ab100db9251, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/3a71fe21617c4b1ab6eb9dcf97d7ae32, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/31184dfe408840958732e3a37a4b80f6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/b37e2a72b6704dd3a40fcb934876161b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1e2d0e3a06354f08a29ec44d041cac1e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47cdf4104a7e4f1d8d87ea14dabbeaae, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/0fca8c811cfa49ed9114ee2e3505065a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/267247875ddc47ee8bce90854b8e1ba3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c7b8a14981074b8f8673d701662049a3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/23b36bd738864418bfc3151b184fef54] to archive 2023-04-27 01:54:49,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:54:49,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/68f5b9c62566414ea7c729ece43a77f0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/68f5b9c62566414ea7c729ece43a77f0 2023-04-27 01:54:49,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/e2d9e8ba8db2440a80a3ce6ccd21e5b3 2023-04-27 01:54:49,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c13bac0836b24282a10c26a35e689944 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c13bac0836b24282a10c26a35e689944 2023-04-27 01:54:49,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/5f103f320c0949c6bae66091312fe297 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/5f103f320c0949c6bae66091312fe297 2023-04-27 01:54:49,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/38bc8e3c4db64b8d93af35e19ad1cf48 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/38bc8e3c4db64b8d93af35e19ad1cf48 2023-04-27 01:54:49,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/be161a8104c9411587529ca30d478d88 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/be161a8104c9411587529ca30d478d88 2023-04-27 01:54:49,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1858cfd62f224cb7aecfd04f0f0cf3a0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1858cfd62f224cb7aecfd04f0f0cf3a0 2023-04-27 01:54:49,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/4d5ad3d87cb6486fb701fa42b9f893f4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/4d5ad3d87cb6486fb701fa42b9f893f4 2023-04-27 01:54:49,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/a5b57e94be9540e68e0c546be3f4f9f2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/a5b57e94be9540e68e0c546be3f4f9f2 2023-04-27 01:54:49,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/6042ce553f7940cbbba6b9681eb89539 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/6042ce553f7940cbbba6b9681eb89539 2023-04-27 01:54:49,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/78253a2366dc4675a28c6c87538442cc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/78253a2366dc4675a28c6c87538442cc 2023-04-27 01:54:49,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/17035dacd68148df8ba639502a261eef to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/17035dacd68148df8ba639502a261eef 2023-04-27 01:54:49,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47ad03f2190d45e38fa25964ce15341c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47ad03f2190d45e38fa25964ce15341c 2023-04-27 01:54:49,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/eb72643e33ae4cb19ea20ab100db9251 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/eb72643e33ae4cb19ea20ab100db9251 2023-04-27 01:54:49,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/3a71fe21617c4b1ab6eb9dcf97d7ae32 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/3a71fe21617c4b1ab6eb9dcf97d7ae32 2023-04-27 01:54:49,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/31184dfe408840958732e3a37a4b80f6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/31184dfe408840958732e3a37a4b80f6 2023-04-27 01:54:49,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/b37e2a72b6704dd3a40fcb934876161b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/b37e2a72b6704dd3a40fcb934876161b 2023-04-27 01:54:49,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1e2d0e3a06354f08a29ec44d041cac1e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/1e2d0e3a06354f08a29ec44d041cac1e 2023-04-27 01:54:49,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47cdf4104a7e4f1d8d87ea14dabbeaae to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/47cdf4104a7e4f1d8d87ea14dabbeaae 2023-04-27 01:54:49,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/0fca8c811cfa49ed9114ee2e3505065a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/0fca8c811cfa49ed9114ee2e3505065a 2023-04-27 01:54:49,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/267247875ddc47ee8bce90854b8e1ba3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/267247875ddc47ee8bce90854b8e1ba3 2023-04-27 01:54:49,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c7b8a14981074b8f8673d701662049a3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/c7b8a14981074b8f8673d701662049a3 2023-04-27 01:54:49,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/23b36bd738864418bfc3151b184fef54 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/23b36bd738864418bfc3151b184fef54 2023-04-27 01:54:49,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/3e584efcf192401abe73f253a48b792e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8b39e429463540d1a89c153204504aac, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/db6e0cda7f1c442684913ff48fdb266d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e3a7d817eae84ec4bd098cdb6a44429d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/48be0368898340a3a77a4defd8ece8fd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8845b3d4ea8942d6b8d3c747d8a7b904, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d598ef38bc824999b6878ed470f7fa63, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/f4e12b7fad5647f2923c88cac15d73ac, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d360140c026a4b0f8089f2471807d088, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9310685781e84fd785d1962b0ce19e0c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/6ad734faf381496aa5a6e9706d5b2ec8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/149b8dc9de774713bed680843e307dee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e5f6f982fbeb4d10a01db48e4785889d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9387d0d92b64482b835b1171f4ece938, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/5f90854c1c8d441288ecb55d3b5bf8ef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/975adf4cc5dd46069a0f2ecc41db4d6a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/b274f1a69ced4a0887a74afd2aa5d78a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/a47631ec47a7404c9ef58097eb6e0d43, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/858da910ba7c4597aca04c49da6aaca4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/c26dfbf8b7814a7e9d89b422a356546b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/4d8139c61deb424887e4838c589fee5f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/543be770996745a4ad12746ca7c66dee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/245abe3e9fec4ba993ec07d82e3456b3] to archive 2023-04-27 01:54:49,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:54:49,630 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/3e584efcf192401abe73f253a48b792e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/3e584efcf192401abe73f253a48b792e 2023-04-27 01:54:49,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8b39e429463540d1a89c153204504aac to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8b39e429463540d1a89c153204504aac 2023-04-27 01:54:49,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/db6e0cda7f1c442684913ff48fdb266d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/db6e0cda7f1c442684913ff48fdb266d 2023-04-27 01:54:49,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e3a7d817eae84ec4bd098cdb6a44429d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e3a7d817eae84ec4bd098cdb6a44429d 2023-04-27 01:54:49,633 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/48be0368898340a3a77a4defd8ece8fd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/48be0368898340a3a77a4defd8ece8fd 2023-04-27 01:54:49,634 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8845b3d4ea8942d6b8d3c747d8a7b904 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8845b3d4ea8942d6b8d3c747d8a7b904 2023-04-27 01:54:49,635 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d598ef38bc824999b6878ed470f7fa63 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d598ef38bc824999b6878ed470f7fa63 2023-04-27 01:54:49,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/f4e12b7fad5647f2923c88cac15d73ac to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/f4e12b7fad5647f2923c88cac15d73ac 2023-04-27 01:54:49,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d360140c026a4b0f8089f2471807d088 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/d360140c026a4b0f8089f2471807d088 2023-04-27 01:54:49,637 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9310685781e84fd785d1962b0ce19e0c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9310685781e84fd785d1962b0ce19e0c 2023-04-27 01:54:49,638 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/6ad734faf381496aa5a6e9706d5b2ec8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/6ad734faf381496aa5a6e9706d5b2ec8 2023-04-27 01:54:49,639 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/149b8dc9de774713bed680843e307dee to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/149b8dc9de774713bed680843e307dee 2023-04-27 01:54:49,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e5f6f982fbeb4d10a01db48e4785889d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/e5f6f982fbeb4d10a01db48e4785889d 2023-04-27 01:54:49,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9387d0d92b64482b835b1171f4ece938 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/9387d0d92b64482b835b1171f4ece938 2023-04-27 01:54:49,641 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/5f90854c1c8d441288ecb55d3b5bf8ef to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/5f90854c1c8d441288ecb55d3b5bf8ef 2023-04-27 01:54:49,642 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/975adf4cc5dd46069a0f2ecc41db4d6a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/975adf4cc5dd46069a0f2ecc41db4d6a 2023-04-27 01:54:49,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/b274f1a69ced4a0887a74afd2aa5d78a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/b274f1a69ced4a0887a74afd2aa5d78a 2023-04-27 01:54:49,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/a47631ec47a7404c9ef58097eb6e0d43 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/a47631ec47a7404c9ef58097eb6e0d43 2023-04-27 01:54:49,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/858da910ba7c4597aca04c49da6aaca4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/858da910ba7c4597aca04c49da6aaca4 2023-04-27 01:54:49,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/c26dfbf8b7814a7e9d89b422a356546b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/c26dfbf8b7814a7e9d89b422a356546b 2023-04-27 01:54:49,646 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/4d8139c61deb424887e4838c589fee5f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/4d8139c61deb424887e4838c589fee5f 2023-04-27 01:54:49,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/543be770996745a4ad12746ca7c66dee to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/543be770996745a4ad12746ca7c66dee 2023-04-27 01:54:49,648 DEBUG [StoreCloser-TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/245abe3e9fec4ba993ec07d82e3456b3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/245abe3e9fec4ba993ec07d82e3456b3 2023-04-27 01:54:49,651 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/recovered.edits/375.seqid, newMaxSeqId=375, maxSeqId=4 2023-04-27 01:54:49,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86. 2023-04-27 01:54:49,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for da5b8d348b0d50311fe8eae652165c86: 2023-04-27 01:54:49,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:49,654 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=da5b8d348b0d50311fe8eae652165c86, regionState=CLOSED 2023-04-27 01:54:49,655 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560489654"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560489654"}]},"ts":"1682560489654"} 2023-04-27 01:54:49,657 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-27 01:54:49,657 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure da5b8d348b0d50311fe8eae652165c86, server=jenkins-hbase4.apache.org,38085,1682560417985 in 1.9060 sec 2023-04-27 01:54:49,658 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-27 01:54:49,658 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=da5b8d348b0d50311fe8eae652165c86, UNASSIGN in 1.9110 sec 2023-04-27 01:54:49,659 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560489658"}]},"ts":"1682560489658"} 2023-04-27 01:54:49,659 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 01:54:49,665 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 01:54:49,666 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9260 sec 2023-04-27 01:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 01:54:49,847 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-27 01:54:49,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 01:54:49,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:49,850 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:49,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-27 01:54:49,851 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:49,855 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:49,858 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/recovered.edits] 2023-04-27 01:54:49,860 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/65eb5335f1c64906bebc26753bb3def7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/65eb5335f1c64906bebc26753bb3def7 2023-04-27 01:54:49,862 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/9c8c2dc9f44f490ea22ff146aa1fa9db to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/A/9c8c2dc9f44f490ea22ff146aa1fa9db 2023-04-27 01:54:49,864 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/14d89230c8c44744bf39d37e426cfa14 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/14d89230c8c44744bf39d37e426cfa14 2023-04-27 01:54:49,865 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/ac6fc73b88c04cd3b129bb18c5fc759a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/B/ac6fc73b88c04cd3b129bb18c5fc759a 2023-04-27 01:54:49,867 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8934075ccc794035a7c590419e2e930d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/8934075ccc794035a7c590419e2e930d 2023-04-27 01:54:49,868 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/ac2df35c74c34b0ebe09949f58a23e89 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/C/ac2df35c74c34b0ebe09949f58a23e89 2023-04-27 01:54:49,870 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/recovered.edits/375.seqid to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86/recovered.edits/375.seqid 2023-04-27 01:54:49,871 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/da5b8d348b0d50311fe8eae652165c86 2023-04-27 01:54:49,871 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:54:49,871 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 01:54:49,872 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-27 01:54:49,875 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042705688889165946f6b33c9ed4d80c0840 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042705688889165946f6b33c9ed4d80c0840 2023-04-27 01:54:49,875 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427112c805f92984dbba0366b43c9c5ca4b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427112c805f92984dbba0366b43c9c5ca4b 2023-04-27 01:54:49,876 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042715db6219d5e74635a5ef1f8c842ad895 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042715db6219d5e74635a5ef1f8c842ad895 2023-04-27 01:54:49,877 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042728772339704049ca803f5d3671644480 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042728772339704049ca803f5d3671644480 2023-04-27 01:54:49,878 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d42d0d6d2904814a19e2a8b3becd2de to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d42d0d6d2904814a19e2a8b3becd2de 2023-04-27 01:54:49,879 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042731377b199c8e400daef3294838f7a580 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042731377b199c8e400daef3294838f7a580 2023-04-27 01:54:49,880 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427351f9bf1a597439eb9333ae00d700621 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427351f9bf1a597439eb9333ae00d700621 2023-04-27 01:54:49,881 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304273faf513314974e76962b89b875ec7a92 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304273faf513314974e76962b89b875ec7a92 2023-04-27 01:54:49,882 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042743200efd0e3f4d698da3f1a4e86456f8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042743200efd0e3f4d698da3f1a4e86456f8 2023-04-27 01:54:49,883 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042749421f0751e448d78ecde1aa8d7205b2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042749421f0751e448d78ecde1aa8d7205b2 2023-04-27 01:54:49,884 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042753e44e0f7f874f5cab4a70c9c22b5c6f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042753e44e0f7f874f5cab4a70c9c22b5c6f 2023-04-27 01:54:49,885 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042773517b9d485f485e8ad58e4dbe833650 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042773517b9d485f485e8ad58e4dbe833650 2023-04-27 01:54:49,886 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042786b96ee7755a4edbb7bbefc7107453f2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042786b96ee7755a4edbb7bbefc7107453f2 2023-04-27 01:54:49,887 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278f191d3595224821a7e30a3ab1bbfb1f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278f191d3595224821a7e30a3ab1bbfb1f 2023-04-27 01:54:49,888 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d7ace99fdb6743bb9cd7df5f76b52e2e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d7ace99fdb6743bb9cd7df5f76b52e2e 2023-04-27 01:54:49,889 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427db614e59af004ee0b9e3356eec4e9d3a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427db614e59af004ee0b9e3356eec4e9d3a 2023-04-27 01:54:49,890 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427dca9a67b711941d597c01aa63a558aa8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427dca9a67b711941d597c01aa63a558aa8 2023-04-27 01:54:49,891 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e486d4a317714bd2a9f8d38ec9aba784 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e486d4a317714bd2a9f8d38ec9aba784 2023-04-27 01:54:49,892 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e5b0fc556ba549fbbfe794c29b66bab1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e5b0fc556ba549fbbfe794c29b66bab1 2023-04-27 01:54:49,893 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e70a7c62e07e432f9e5f8b23d35794a1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e70a7c62e07e432f9e5f8b23d35794a1 2023-04-27 01:54:49,894 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 01:54:49,895 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:49,897 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 01:54:49,899 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 01:54:49,900 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:49,900 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 01:54:49,900 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682560489900"}]},"ts":"9223372036854775807"} 2023-04-27 01:54:49,902 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 01:54:49,902 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => da5b8d348b0d50311fe8eae652165c86, NAME => 'TestAcidGuarantees,,1682560455300.da5b8d348b0d50311fe8eae652165c86.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 01:54:49,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 01:54:49,902 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682560489902"}]},"ts":"9223372036854775807"} 2023-04-27 01:54:49,903 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 01:54:49,905 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:49,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-04-27 01:54:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-27 01:54:49,952 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-27 01:54:49,964 INFO [Listener at localhost/42495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 229), OpenFileDescriptor=514 (was 542), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=329 (was 405), ProcessCount=170 (was 170), AvailableMemoryMB=12038 (was 12068) 2023-04-27 01:54:49,975 INFO [Listener at localhost/42495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=514, MaxFileDescriptor=60000, SystemLoadAverage=329, ProcessCount=170, AvailableMemoryMB=12038 2023-04-27 01:54:49,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:54:49,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:54:49,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 01:54:49,979 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 01:54:49,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-27 01:54:49,980 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 01:54:49,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 01:54:49,981 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:49,981 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604 empty. 2023-04-27 01:54:49,982 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:49,982 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:54:50,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 01:54:50,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 01:54:50,389 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:54:50,391 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6f2bfc2234dd9d4cdcd80dce1fbcb604, NAME => 'TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp 2023-04-27 01:54:50,401 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:54:50,401 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6f2bfc2234dd9d4cdcd80dce1fbcb604, disabling compactions & flushes 2023-04-27 01:54:50,401 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,401 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,401 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. after waiting 0 ms 2023-04-27 01:54:50,401 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,401 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,401 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:50,403 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 01:54:50,404 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560490404"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560490404"}]},"ts":"1682560490404"} 2023-04-27 01:54:50,405 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 01:54:50,406 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 01:54:50,406 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560490406"}]},"ts":"1682560490406"} 2023-04-27 01:54:50,407 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 01:54:50,409 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2bfc2234dd9d4cdcd80dce1fbcb604, ASSIGN}] 2023-04-27 01:54:50,411 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2bfc2234dd9d4cdcd80dce1fbcb604, ASSIGN 2023-04-27 01:54:50,411 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2bfc2234dd9d4cdcd80dce1fbcb604, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=false 2023-04-27 01:54:50,563 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=6f2bfc2234dd9d4cdcd80dce1fbcb604, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:50,563 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560490563"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560490563"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560490563"}]},"ts":"1682560490563"} 2023-04-27 01:54:50,564 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:54:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 01:54:50,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6f2bfc2234dd9d4cdcd80dce1fbcb604, NAME => 'TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:54:50,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:54:50,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,721 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,722 DEBUG [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A 2023-04-27 01:54:50,722 DEBUG [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A 2023-04-27 01:54:50,722 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:50,723 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f2bfc2234dd9d4cdcd80dce1fbcb604 columnFamilyName A 2023-04-27 01:54:50,723 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.HStore(310): Store=6f2bfc2234dd9d4cdcd80dce1fbcb604/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:50,723 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,724 DEBUG [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B 2023-04-27 01:54:50,724 DEBUG [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B 2023-04-27 01:54:50,725 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:50,725 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f2bfc2234dd9d4cdcd80dce1fbcb604 columnFamilyName B 2023-04-27 01:54:50,725 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.HStore(310): Store=6f2bfc2234dd9d4cdcd80dce1fbcb604/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:50,725 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,726 DEBUG [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C 2023-04-27 01:54:50,726 DEBUG [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C 2023-04-27 01:54:50,726 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:54:50,726 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f2bfc2234dd9d4cdcd80dce1fbcb604 columnFamilyName C 2023-04-27 01:54:50,727 INFO [StoreOpener-6f2bfc2234dd9d4cdcd80dce1fbcb604-1] regionserver.HStore(310): Store=6f2bfc2234dd9d4cdcd80dce1fbcb604/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:54:50,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:54:50,730 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:50,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:54:50,732 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6f2bfc2234dd9d4cdcd80dce1fbcb604; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63811243, jitterRate=-0.0491383820772171}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:54:50,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:50,733 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., pid=30, masterSystemTime=1682560490716 2023-04-27 01:54:50,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,734 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:50,734 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=6f2bfc2234dd9d4cdcd80dce1fbcb604, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:50,735 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560490734"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560490734"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560490734"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560490734"}]},"ts":"1682560490734"} 2023-04-27 01:54:50,737 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-27 01:54:50,737 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 in 171 msec 2023-04-27 01:54:50,738 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-27 01:54:50,738 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2bfc2234dd9d4cdcd80dce1fbcb604, ASSIGN in 328 msec 2023-04-27 01:54:50,739 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 01:54:50,739 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560490739"}]},"ts":"1682560490739"} 2023-04-27 01:54:50,739 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 01:54:50,743 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 01:54:50,744 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 766 msec 2023-04-27 01:54:51,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 01:54:51,084 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-27 01:54:51,085 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x23083cf2 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,090 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52446f97, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,091 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,092 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,093 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 01:54:51,095 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 01:54:51,096 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x47772cff to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,099 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25fa674f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,100 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x53921272 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,105 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7cdc2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,105 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x13544701 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,112 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5096c69b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,113 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x10b79c2c to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,116 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@216aad74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,116 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5a81fa to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,120 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76203061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,120 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x3b39d5b8 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,124 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c6d5986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,124 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x44f9bd75 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,131 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31182447, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,131 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x5e5b1c14 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,134 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574000bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,134 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x4f29c1c4 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,141 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4994df89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,142 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x681ace6c to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:54:51,146 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69bc154a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:54:51,152 DEBUG [hconnection-0x2e6b57c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,152 DEBUG [hconnection-0x1aba3be8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,152 DEBUG [hconnection-0x65ee6fde-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,152 DEBUG [hconnection-0x6ebe231b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,153 DEBUG [hconnection-0xe4644fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,154 DEBUG [hconnection-0x565713d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,155 DEBUG [hconnection-0x55367947-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,155 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,155 DEBUG [hconnection-0x30b2c73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,157 DEBUG [hconnection-0x6095ddd4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,157 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,157 DEBUG [hconnection-0x5658fb6f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:54:51,157 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,159 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38672, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,160 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,159 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,158 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,162 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:51,167 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,168 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38718, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:54:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:51,177 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:51,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:51,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:51,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,180 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:51,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:51,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:51,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:54:51,188 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:51,189 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:51,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:51,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:51,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:51,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:51,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:51,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b8e9343[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:51,195 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:51,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:51,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:51,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:51,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:51,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:51,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:51,200 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:51,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:51,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:51,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:51,200 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:51,201 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,201 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,201 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/cc0cf281ea7e4795961f514e9cdb68cc 2023-04-27 01:54:51,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,203 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,209 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,209 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,209 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:51,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:51,210 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:51,210 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:51,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560551206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560551206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560551210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560551210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560551211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,310 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:51,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:51,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:51,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560551312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560551313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560551314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560551314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560551315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:51,325 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:51,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:51,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:51,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:51,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:51,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:51,329 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,329 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:51,329 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:51,329 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,332 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:51,334 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:51,334 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51d26aa1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:51,335 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,335 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:51,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:51,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:51,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:51,337 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:51,340 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:51,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:51,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:51,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:51,340 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:51,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,342 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:51,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,349 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:51,350 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:51,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:51,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,450 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:51,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:51,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:51,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:51,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:51,453 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:51,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:51,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:51,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,455 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,455 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,455 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:51,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:51,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,456 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:51,457 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:51,457 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:51,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,458 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,458 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e19ef53[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:51,460 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:51,460 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,461 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:51,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,461 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:51,461 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:51,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,464 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:51,466 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:51,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:51,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:51,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:51,466 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:51,468 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,468 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,468 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,468 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,472 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:51,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:51,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:51,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,473 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:51,473 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:51,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560551514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560551515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560551517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560551518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560551518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,573 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:51,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:51,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:51,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:51,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:51,577 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:51,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:51,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:51,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,578 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:51,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:51,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:51,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:51,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,585 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ba4ffa[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:51,587 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:51,587 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,588 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,588 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:51,588 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:51,588 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:51,589 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,589 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:51,590 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:51,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:51,593 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:51,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:51,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:51,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:51,593 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:51,594 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,594 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,594 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,596 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,599 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:51,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:51,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,600 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,600 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,600 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,600 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:51,600 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:51,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/f1a049bfee774c329128560498fd5ae1 2023-04-27 01:54:51,700 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:51,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:51,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:51,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:51,703 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:51,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:51,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:51,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,705 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,705 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,705 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:51,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:51,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:51,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:51,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,706 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:51,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:51,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:51,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:51,708 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,708 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7990eb1e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:51,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:51,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:51,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:51,713 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:51,716 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:51,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:51,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:51,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:51,716 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:51,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,718 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:51,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:51,722 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,722 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:51,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,723 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560551817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560551820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560551820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,823 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:51,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:51,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:51,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560551822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560551822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:51,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:51,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:51,827 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:51,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:51,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:51,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,829 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:51,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:51,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:51,830 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,831 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:51,831 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:51,831 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:51,832 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:51,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b5501ef[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:51,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:51,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:51,836 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,836 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:51,838 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:51,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:51,841 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:51,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:51,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:51,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:51,841 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:51,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,843 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:51,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:51,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:51,847 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,848 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:51,848 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,948 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:51,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:51,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:51,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:51,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:51,951 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:51,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:51,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:51,952 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,952 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,953 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:51,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:51,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:51,955 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,955 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:51,955 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:51,955 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:51,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7af712a1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:51,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:51,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,963 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:51,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:51,965 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:51,965 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,965 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,965 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:51,967 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:51,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:51,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:51,967 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:51,967 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:51,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:51,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,971 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,972 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:51,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:51,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:51,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:51,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:51,975 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:51,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:51,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:51,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:51,975 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:51,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:51,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,975 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:51,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:51,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/2273fdbb1a5448ac9090b25bdad1b7cc 2023-04-27 01:54:52,075 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:52,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:52,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:52,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:52,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:52,079 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:52,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:52,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:52,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:52,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,081 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:52,081 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:52,081 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,081 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/cc0cf281ea7e4795961f514e9cdb68cc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/cc0cf281ea7e4795961f514e9cdb68cc 2023-04-27 01:54:52,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:52,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:52,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:52,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:52,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:52,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:52,083 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:52,083 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:52,083 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:52,083 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:52,085 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:52,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/cc0cf281ea7e4795961f514e9cdb68cc, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 01:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/f1a049bfee774c329128560498fd5ae1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f1a049bfee774c329128560498fd5ae1 2023-04-27 01:54:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:52,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a0579a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:52,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:52,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f1a049bfee774c329128560498fd5ae1, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 01:54:52,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/2273fdbb1a5448ac9090b25bdad1b7cc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/2273fdbb1a5448ac9090b25bdad1b7cc 2023-04-27 01:54:52,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:52,093 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,093 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:52,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:52,098 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:52,098 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:52,098 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:52,098 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:52,098 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:52,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:52,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:52,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,104 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/2273fdbb1a5448ac9090b25bdad1b7cc, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:52,106 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:52,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:52,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:52,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:52,107 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:52,107 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:52,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 919ms, sequenceid=13, compaction requested=false 2023-04-27 01:54:52,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:52,207 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:52,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:52,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:52,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:52,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:52,210 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:52,210 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:52,211 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:52,211 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:52,211 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,211 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,213 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:52,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:52,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,213 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:52,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:52,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:52,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:52,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:52,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:52,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:52,215 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:52,215 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:52,215 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:52,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:52,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:52,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560552330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560552330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560552332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560552333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560552333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560552434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560552434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560552435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560552436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560552436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,624 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/1235b0df27c7436dbb12099e4664751d 2023-04-27 01:54:52,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560552638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560552638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,644 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/9fba5212580d4ed6a7be6dc8cd087cf5 2023-04-27 01:54:52,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560552643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560552643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560552644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,658 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/b83dcda6a3f447b1bd8fb10f38c4dc0b 2023-04-27 01:54:52,662 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/1235b0df27c7436dbb12099e4664751d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/1235b0df27c7436dbb12099e4664751d 2023-04-27 01:54:52,665 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/1235b0df27c7436dbb12099e4664751d, entries=150, sequenceid=33, filesize=11.6 K 2023-04-27 01:54:52,666 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/9fba5212580d4ed6a7be6dc8cd087cf5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/9fba5212580d4ed6a7be6dc8cd087cf5 2023-04-27 01:54:52,670 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/9fba5212580d4ed6a7be6dc8cd087cf5, entries=150, sequenceid=33, filesize=11.6 K 2023-04-27 01:54:52,672 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/b83dcda6a3f447b1bd8fb10f38c4dc0b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b83dcda6a3f447b1bd8fb10f38c4dc0b 2023-04-27 01:54:52,675 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b83dcda6a3f447b1bd8fb10f38c4dc0b, entries=150, sequenceid=33, filesize=11.6 K 2023-04-27 01:54:52,676 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 461ms, sequenceid=33, compaction requested=false 2023-04-27 01:54:52,676 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:52,676 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:52,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1696898b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:52,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:52,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,681 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:52,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:52,683 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:52,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,683 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:52,683 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:52,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:52,686 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:52,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:52,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:52,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:52,686 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:52,688 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,688 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,688 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:52,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:52,689 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:52,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:52,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:52,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:52,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,693 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:52,693 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:52,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,793 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:52,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:52,797 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:52,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:52,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:52,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:52,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,798 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:52,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:52,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:52,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:52,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:52,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:52,800 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:52,800 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:52,801 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:54:52,801 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:52,801 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:52,801 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:52,801 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:52,801 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:52,801 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:52,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:52,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:52,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560552955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560552957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560552958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560552958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:52,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560552959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560553060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560553060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560553061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560553062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560553062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,211 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/a65f2d01dd314eddb56636a0ca5695ba 2023-04-27 01:54:53,224 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/b2f2716892a04bdd9874dbd2b4f915e3 2023-04-27 01:54:53,235 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/c78680a039ae434b8ff9bc4ca1bf150f 2023-04-27 01:54:53,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/a65f2d01dd314eddb56636a0ca5695ba as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/a65f2d01dd314eddb56636a0ca5695ba 2023-04-27 01:54:53,243 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/a65f2d01dd314eddb56636a0ca5695ba, entries=150, sequenceid=45, filesize=11.6 K 2023-04-27 01:54:53,244 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/b2f2716892a04bdd9874dbd2b4f915e3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b2f2716892a04bdd9874dbd2b4f915e3 2023-04-27 01:54:53,248 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b2f2716892a04bdd9874dbd2b4f915e3, entries=150, sequenceid=45, filesize=11.6 K 2023-04-27 01:54:53,249 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/c78680a039ae434b8ff9bc4ca1bf150f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c78680a039ae434b8ff9bc4ca1bf150f 2023-04-27 01:54:53,253 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c78680a039ae434b8ff9bc4ca1bf150f, entries=150, sequenceid=45, filesize=11.6 K 2023-04-27 01:54:53,254 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 454ms, sequenceid=45, compaction requested=true 2023-04-27 01:54:53,254 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:53,254 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:53,255 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55fb8cf[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:53,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:53,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:53,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:53,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-27 01:54:53,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560492954, max timestamp=1682560493264Num uniques 859; 2023-04-27 01:54:53,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:54:53,266 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560492954, max timestamp=1682560493264Num uniques 859; 2023-04-27 01:54:53,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:54:53,266 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:53,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:53,268 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:53,268 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:53,268 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:53,268 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:53,268 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:53,271 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:53,271 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,271 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:53,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:53,276 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:53,276 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:53,276 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:53,276 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:53,276 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:53,278 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,278 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,278 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,279 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/ab7f853af104407a91a019605f13dcfe 2023-04-27 01:54:53,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:53,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:53,285 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:53,285 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:53,284 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,285 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,285 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,285 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,285 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,285 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/aa5c765d083e4004974da993863b6a8d 2023-04-27 01:54:53,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560553285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560553290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560553291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560553292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560553293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,385 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:53,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:53,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:53,388 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:53,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:53,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:53,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,390 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,391 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:53,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:53,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:53,392 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,392 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:53,392 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:53,392 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,394 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560553392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560553397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560553397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560553398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560553398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:53,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477fc0df[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:53,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:53,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:53,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:53,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:53,401 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,401 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:53,403 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,403 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:53,406 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:53,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:53,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:53,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:53,406 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:53,407 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,407 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,408 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,408 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,409 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,411 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,411 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,412 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,412 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,412 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,412 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:53,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:53,412 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,412 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,412 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,412 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,412 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:53,412 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:53,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,512 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:53,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:53,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:53,516 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:53,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:53,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:53,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,517 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,518 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,518 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:53,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:53,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:53,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:53,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:53,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:53,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d09532d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:53,523 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:53,524 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,524 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:53,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:53,526 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:53,528 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:53,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:53,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:53,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:53,528 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:53,529 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,529 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,529 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,530 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:53,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:53,534 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,534 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:53,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,533 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,534 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:53,534 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,534 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:53,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560553597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560553600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560553600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560553600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560553600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,634 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:53,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:53,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:53,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:53,638 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:53,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:53,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:53,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,639 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,639 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,639 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,639 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:53,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:53,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:53,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:53,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:53,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:53,642 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:53,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:53,645 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:53,645 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff15dd6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:53,646 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:53,646 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,646 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:53,648 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:53,650 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:53,650 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:53,651 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:53,651 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:53,651 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:53,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:53,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:53,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,654 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,658 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:53,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:53,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:53,658 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,658 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:53,658 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:53,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:53,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:53,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/f83ca41edc7a4b30831ab71da05a89ba 2023-04-27 01:54:53,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/ab7f853af104407a91a019605f13dcfe as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/ab7f853af104407a91a019605f13dcfe 2023-04-27 01:54:53,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/ab7f853af104407a91a019605f13dcfe, entries=150, sequenceid=68, filesize=11.6 K 2023-04-27 01:54:53,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/aa5c765d083e4004974da993863b6a8d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/aa5c765d083e4004974da993863b6a8d 2023-04-27 01:54:53,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/aa5c765d083e4004974da993863b6a8d, entries=150, sequenceid=68, filesize=11.6 K 2023-04-27 01:54:53,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/f83ca41edc7a4b30831ab71da05a89ba as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f83ca41edc7a4b30831ab71da05a89ba 2023-04-27 01:54:53,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f83ca41edc7a4b30831ab71da05a89ba, entries=150, sequenceid=68, filesize=11.6 K 2023-04-27 01:54:53,720 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~203.97 KB/208864, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 455ms, sequenceid=68, compaction requested=true 2023-04-27 01:54:53,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:53,720 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:53,720 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:53,720 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:54:53,720 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:54:53,720 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/A is initiating minor compaction (all files) 2023-04-27 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:54:53,721 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/A in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,721 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/B is initiating minor compaction (all files) 2023-04-27 01:54:53,722 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/cc0cf281ea7e4795961f514e9cdb68cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/1235b0df27c7436dbb12099e4664751d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/a65f2d01dd314eddb56636a0ca5695ba, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/ab7f853af104407a91a019605f13dcfe] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=46.4 K 2023-04-27 01:54:53,722 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/B in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,722 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f1a049bfee774c329128560498fd5ae1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/9fba5212580d4ed6a7be6dc8cd087cf5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b2f2716892a04bdd9874dbd2b4f915e3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/aa5c765d083e4004974da993863b6a8d] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=46.4 K 2023-04-27 01:54:53,722 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cc0cf281ea7e4795961f514e9cdb68cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682560491180 2023-04-27 01:54:53,722 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f1a049bfee774c329128560498fd5ae1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682560491180 2023-04-27 01:54:53,722 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 1235b0df27c7436dbb12099e4664751d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682560491204 2023-04-27 01:54:53,723 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9fba5212580d4ed6a7be6dc8cd087cf5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682560491204 2023-04-27 01:54:53,723 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a65f2d01dd314eddb56636a0ca5695ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682560492329 2023-04-27 01:54:53,723 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b2f2716892a04bdd9874dbd2b4f915e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682560492329 2023-04-27 01:54:53,723 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting ab7f853af104407a91a019605f13dcfe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560492954 2023-04-27 01:54:53,723 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting aa5c765d083e4004974da993863b6a8d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560492954 2023-04-27 01:54:53,731 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#A#compaction#204 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:53,732 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#B#compaction#205 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:53,748 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/c79f9245cd1e4af781277b1acd139384 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c79f9245cd1e4af781277b1acd139384 2023-04-27 01:54:53,752 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/A of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into c79f9245cd1e4af781277b1acd139384(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:53,752 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:53,752 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/A, priority=12, startTime=1682560493720; duration=0sec 2023-04-27 01:54:53,752 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:53,752 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:54:53,754 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:54:53,754 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/C is initiating minor compaction (all files) 2023-04-27 01:54:53,754 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/C in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,754 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/2273fdbb1a5448ac9090b25bdad1b7cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b83dcda6a3f447b1bd8fb10f38c4dc0b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c78680a039ae434b8ff9bc4ca1bf150f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f83ca41edc7a4b30831ab71da05a89ba] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=46.4 K 2023-04-27 01:54:53,754 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2273fdbb1a5448ac9090b25bdad1b7cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682560491180 2023-04-27 01:54:53,754 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b83dcda6a3f447b1bd8fb10f38c4dc0b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682560491204 2023-04-27 01:54:53,755 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c78680a039ae434b8ff9bc4ca1bf150f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682560492329 2023-04-27 01:54:53,755 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f83ca41edc7a4b30831ab71da05a89ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560492954 2023-04-27 01:54:53,759 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:53,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:53,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:53,770 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:53,770 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,770 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:53,770 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:53,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,773 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:53,773 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:53,775 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,775 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:53,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:53,777 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#C#compaction#206 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:53,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:53,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:53,780 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:53,780 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:53,780 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:53,788 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:53,789 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:53,789 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:53,789 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:53,789 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:53,789 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:53,789 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:53,789 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:53,789 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:53,816 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/8174e527f8a046b1be220e064cdfc70f 2023-04-27 01:54:53,838 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/8da16e7bc6654cd992297869db0bb628 2023-04-27 01:54:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:53,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:53,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560553906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560553906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560553907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560553908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:53,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:53,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560553909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560554010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560554010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560554010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560554012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560554012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,148 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/2c199e40a8a84418b0f2fbc0a2e80e81 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2c199e40a8a84418b0f2fbc0a2e80e81 2023-04-27 01:54:54,152 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/B of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 2c199e40a8a84418b0f2fbc0a2e80e81(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:54,152 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:54,152 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/B, priority=12, startTime=1682560493720; duration=0sec 2023-04-27 01:54:54,152 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:54,209 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/90b56aa9b31c4859b396b1aa4362d55c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/90b56aa9b31c4859b396b1aa4362d55c 2023-04-27 01:54:54,214 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/C of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 90b56aa9b31c4859b396b1aa4362d55c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:54,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,215 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:54,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560554213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,215 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/C, priority=12, startTime=1682560493720; duration=0sec 2023-04-27 01:54:54,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,215 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:54,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560554214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560554214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560554214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560554216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,254 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/b05ff8c78cfc461e92131a1aad51e757 2023-04-27 01:54:54,259 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/8174e527f8a046b1be220e064cdfc70f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8174e527f8a046b1be220e064cdfc70f 2023-04-27 01:54:54,262 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8174e527f8a046b1be220e064cdfc70f, entries=150, sequenceid=89, filesize=11.6 K 2023-04-27 01:54:54,263 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/8da16e7bc6654cd992297869db0bb628 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8da16e7bc6654cd992297869db0bb628 2023-04-27 01:54:54,268 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8da16e7bc6654cd992297869db0bb628, entries=150, sequenceid=89, filesize=11.6 K 2023-04-27 01:54:54,271 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/b05ff8c78cfc461e92131a1aad51e757 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b05ff8c78cfc461e92131a1aad51e757 2023-04-27 01:54:54,275 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b05ff8c78cfc461e92131a1aad51e757, entries=150, sequenceid=89, filesize=11.6 K 2023-04-27 01:54:54,275 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 486ms, sequenceid=89, compaction requested=false 2023-04-27 01:54:54,276 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:54,276 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:54,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,278 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:54,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77900527[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:54,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:54,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:54,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:54,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,282 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:54,282 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:54,282 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:54,285 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:54,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:54,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:54,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:54,285 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:54,288 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:54,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:54,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:54,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:54,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:54,294 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:54,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:54,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:54,295 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:54,295 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:54,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:54,395 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:54,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:54,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:54,398 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:54,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:54,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:54,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:54,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,400 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:54,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:54,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:54,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:54,402 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:54:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:54,403 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:54,421 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/2d35d8b20f394d9b953ba18429bef6d8 2023-04-27 01:54:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:54,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:54,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560554527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560554528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560494526, max timestamp=1682560494530Num uniques 859; 2023-04-27 01:54:54,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682560494526, max timestamp=1682560494530Num uniques 859; 2023-04-27 01:54:54,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:54,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 01:54:54,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560494526, max timestamp=1682560494530Num uniques 859; 2023-04-27 01:54:54,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:54,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560554545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560554546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560554549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560554633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560554635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560554650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560554650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560554652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560554836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560554838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,842 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/b1cb7fef8f744991bd290bf235860a23 2023-04-27 01:54:54,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,853 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/f4c8ff4d70344eca8798dbad8c65a8f1 2023-04-27 01:54:54,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560554851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560554854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,858 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/2d35d8b20f394d9b953ba18429bef6d8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/2d35d8b20f394d9b953ba18429bef6d8 2023-04-27 01:54:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560554857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,862 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/2d35d8b20f394d9b953ba18429bef6d8, entries=150, sequenceid=103, filesize=11.6 K 2023-04-27 01:54:54,862 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/b1cb7fef8f744991bd290bf235860a23 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b1cb7fef8f744991bd290bf235860a23 2023-04-27 01:54:54,866 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b1cb7fef8f744991bd290bf235860a23, entries=150, sequenceid=103, filesize=11.6 K 2023-04-27 01:54:54,866 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/f4c8ff4d70344eca8798dbad8c65a8f1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f4c8ff4d70344eca8798dbad8c65a8f1 2023-04-27 01:54:54,870 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f4c8ff4d70344eca8798dbad8c65a8f1, entries=150, sequenceid=103, filesize=11.6 K 2023-04-27 01:54:54,870 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=123.00 KB/125950 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 468ms, sequenceid=103, compaction requested=true 2023-04-27 01:54:54,870 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:54,870 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:54,872 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e505a32[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:54,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:54,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,876 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:54,877 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,877 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:54,878 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:54,880 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:54,880 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:54,880 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:54,880 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:54,880 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:54,882 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,882 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:54,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,882 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:54,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:54,882 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:54,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:54,886 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:54,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:54,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:54,887 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:54,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:54,887 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:54,987 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:54,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:54,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:54,990 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:54,990 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:54,990 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:54,990 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:54,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,991 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,998 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:54,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:54,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:54,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:54,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:54,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:55,000 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=123.00 KB heapSize=345.53 KB 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:55,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:55,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:55,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:55,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560555154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560555155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560555157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560555157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560555162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,185 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 01:54:55,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560555259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560555259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,408 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/8d9d09de27b24eb49a563a97cb6d5bb4 2023-04-27 01:54:55,418 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/d9fa4861df8c4595893518ceb318e265 2023-04-27 01:54:55,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560555461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560555462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560555660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560555660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560555671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560555766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:55,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560555766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,830 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/01b44cf2b52044f3ba93ec7e403d1263 2023-04-27 01:54:55,835 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/8d9d09de27b24eb49a563a97cb6d5bb4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8d9d09de27b24eb49a563a97cb6d5bb4 2023-04-27 01:54:55,839 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8d9d09de27b24eb49a563a97cb6d5bb4, entries=150, sequenceid=137, filesize=11.8 K 2023-04-27 01:54:55,840 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/d9fa4861df8c4595893518ceb318e265 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d9fa4861df8c4595893518ceb318e265 2023-04-27 01:54:55,844 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d9fa4861df8c4595893518ceb318e265, entries=150, sequenceid=137, filesize=11.8 K 2023-04-27 01:54:55,844 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/01b44cf2b52044f3ba93ec7e403d1263 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/01b44cf2b52044f3ba93ec7e403d1263 2023-04-27 01:54:55,848 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/01b44cf2b52044f3ba93ec7e403d1263, entries=150, sequenceid=137, filesize=11.8 K 2023-04-27 01:54:55,848 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~345.48 KB/353776, currentSize=60.38 KB/61830 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 848ms, sequenceid=137, compaction requested=true 2023-04-27 01:54:55,849 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:55,849 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:55,850 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a7f8ee[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:55,852 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:55,852 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,853 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:55,853 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:55,853 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,853 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:55,854 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:55,857 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:55,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:55,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:55,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:55,857 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:55,860 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,860 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:55,860 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:55,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:55,861 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:55,862 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,867 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:55,867 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,867 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:55,867 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:55,867 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:55,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:55,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:55,867 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:55,868 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,868 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,868 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:55,868 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,868 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:55,868 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:55,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:55,968 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:55,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:55,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:55,971 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:55,971 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:55,971 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:55,971 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:55,972 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,972 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:55,973 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:55,973 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:55,973 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:55,973 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:55,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:55,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:55,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:55,974 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:55,975 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:55,975 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:55,982 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/94b959930bae4caba5694efca6eab200 2023-04-27 01:54:55,991 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/90b961c41b4b46aa89b8a371c7f43a91 2023-04-27 01:54:56,003 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/e6eb8a1373de4e8aaccfa8c73e1adcf7 2023-04-27 01:54:56,007 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/94b959930bae4caba5694efca6eab200 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/94b959930bae4caba5694efca6eab200 2023-04-27 01:54:56,010 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/94b959930bae4caba5694efca6eab200, entries=150, sequenceid=149, filesize=11.8 K 2023-04-27 01:54:56,011 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/90b961c41b4b46aa89b8a371c7f43a91 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/90b961c41b4b46aa89b8a371c7f43a91 2023-04-27 01:54:56,015 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/90b961c41b4b46aa89b8a371c7f43a91, entries=150, sequenceid=149, filesize=11.8 K 2023-04-27 01:54:56,015 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/e6eb8a1373de4e8aaccfa8c73e1adcf7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e6eb8a1373de4e8aaccfa8c73e1adcf7 2023-04-27 01:54:56,019 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e6eb8a1373de4e8aaccfa8c73e1adcf7, entries=150, sequenceid=149, filesize=11.8 K 2023-04-27 01:54:56,019 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 44ms, sequenceid=149, compaction requested=true 2023-04-27 01:54:56,019 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:56,019 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,019 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:56,019 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:56,019 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:56,022 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c3d0fe1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:56,051 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:56,051 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,053 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:56,053 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,053 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:56,055 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:56,058 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:56,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:56,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:56,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:56,058 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:56,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,061 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,062 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,064 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,069 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:56,069 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:56,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:56,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,169 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:56,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:56,173 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:56,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:56,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:56,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:56,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:56,179 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bf39e7c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:56,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:56,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:56,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:56,184 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:56,187 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:56,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:56,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:56,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:56,187 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:56,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:56,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:56,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:56,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,193 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:56,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,193 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:56,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:56,281 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:54:56,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:56,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:56,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:56,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:56,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:56,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:56,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/9efaefbfccc84fca8daff24ac01545be 2023-04-27 01:54:56,293 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:56,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:56,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:56,296 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:56,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:56,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:56,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:56,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:56,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:56,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:56,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:56,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/22698dc7806948409cb2138921d732b6 2023-04-27 01:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,308 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17becefa[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:56,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:56,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:56,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:56,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/e8737ce7f013445a91e0658930fecbee 2023-04-27 01:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:56,317 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:56,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:56,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:56,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:56,317 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:56,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/9efaefbfccc84fca8daff24ac01545be as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9efaefbfccc84fca8daff24ac01545be 2023-04-27 01:54:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,322 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9efaefbfccc84fca8daff24ac01545be, entries=150, sequenceid=160, filesize=11.8 K 2023-04-27 01:54:56,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/22698dc7806948409cb2138921d732b6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/22698dc7806948409cb2138921d732b6 2023-04-27 01:54:56,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560496309, max timestamp=1682560496326Num uniques 859; 2023-04-27 01:54:56,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:56,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560496309, max timestamp=1682560496326Num uniques 859; 2023-04-27 01:54:56,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:56,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560496309, max timestamp=1682560496326Num uniques 859; 2023-04-27 01:54:56,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:54:56,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:56,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:56,330 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,331 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,331 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,331 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,331 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,331 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:56,331 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:56,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,331 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,332 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/22698dc7806948409cb2138921d732b6, entries=100, sequenceid=160, filesize=9.4 K 2023-04-27 01:54:56,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/e8737ce7f013445a91e0658930fecbee as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e8737ce7f013445a91e0658930fecbee 2023-04-27 01:54:56,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e8737ce7f013445a91e0658930fecbee, entries=100, sequenceid=160, filesize=9.4 K 2023-04-27 01:54:56,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=53.67 KB/54960 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 58ms, sequenceid=160, compaction requested=true 2023-04-27 01:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:56,339 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:54:56,339 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:56,341 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71900 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:54:56,341 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69510 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:54:56,341 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/A is initiating minor compaction (all files) 2023-04-27 01:54:56,341 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/B is initiating minor compaction (all files) 2023-04-27 01:54:56,341 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/A in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,341 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/B in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,341 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c79f9245cd1e4af781277b1acd139384, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8174e527f8a046b1be220e064cdfc70f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/2d35d8b20f394d9b953ba18429bef6d8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8d9d09de27b24eb49a563a97cb6d5bb4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/94b959930bae4caba5694efca6eab200, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9efaefbfccc84fca8daff24ac01545be] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=70.2 K 2023-04-27 01:54:56,341 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2c199e40a8a84418b0f2fbc0a2e80e81, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8da16e7bc6654cd992297869db0bb628, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b1cb7fef8f744991bd290bf235860a23, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d9fa4861df8c4595893518ceb318e265, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/90b961c41b4b46aa89b8a371c7f43a91, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/22698dc7806948409cb2138921d732b6] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=67.9 K 2023-04-27 01:54:56,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,341 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c79f9245cd1e4af781277b1acd139384, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560492954 2023-04-27 01:54:56,342 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2c199e40a8a84418b0f2fbc0a2e80e81, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560492954 2023-04-27 01:54:56,342 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8174e527f8a046b1be220e064cdfc70f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682560493270 2023-04-27 01:54:56,342 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8da16e7bc6654cd992297869db0bb628, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682560493270 2023-04-27 01:54:56,342 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2d35d8b20f394d9b953ba18429bef6d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682560493906 2023-04-27 01:54:56,342 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b1cb7fef8f744991bd290bf235860a23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682560493906 2023-04-27 01:54:56,343 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8d9d09de27b24eb49a563a97cb6d5bb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1682560494541 2023-04-27 01:54:56,343 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d9fa4861df8c4595893518ceb318e265, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1682560494541 2023-04-27 01:54:56,343 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 94b959930bae4caba5694efca6eab200, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682560495146 2023-04-27 01:54:56,343 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 90b961c41b4b46aa89b8a371c7f43a91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682560495146 2023-04-27 01:54:56,343 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9efaefbfccc84fca8daff24ac01545be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682560496276 2023-04-27 01:54:56,344 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 22698dc7806948409cb2138921d732b6, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682560496276 2023-04-27 01:54:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:56,349 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=161.16 KB 2023-04-27 01:54:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:54:56,370 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#B#compaction#223 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:56,380 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#A#compaction#224 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:56,402 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/fa9d04a68fce48cca8e853332837e4a4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/fa9d04a68fce48cca8e853332837e4a4 2023-04-27 01:54:56,407 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/A of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into fa9d04a68fce48cca8e853332837e4a4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:56,407 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:56,407 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/A, priority=10, startTime=1682560496339; duration=0sec 2023-04-27 01:54:56,407 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:54:56,407 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:54:56,409 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69510 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:54:56,409 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/C is initiating minor compaction (all files) 2023-04-27 01:54:56,409 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/C in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,410 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/90b56aa9b31c4859b396b1aa4362d55c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b05ff8c78cfc461e92131a1aad51e757, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f4c8ff4d70344eca8798dbad8c65a8f1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/01b44cf2b52044f3ba93ec7e403d1263, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e6eb8a1373de4e8aaccfa8c73e1adcf7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e8737ce7f013445a91e0658930fecbee] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=67.9 K 2023-04-27 01:54:56,410 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 90b56aa9b31c4859b396b1aa4362d55c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560492954 2023-04-27 01:54:56,410 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b05ff8c78cfc461e92131a1aad51e757, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682560493270 2023-04-27 01:54:56,411 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f4c8ff4d70344eca8798dbad8c65a8f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682560493906 2023-04-27 01:54:56,411 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 01b44cf2b52044f3ba93ec7e403d1263, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1682560494541 2023-04-27 01:54:56,411 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e6eb8a1373de4e8aaccfa8c73e1adcf7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682560495146 2023-04-27 01:54:56,412 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e8737ce7f013445a91e0658930fecbee, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682560496276 2023-04-27 01:54:56,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560556408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560556412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,424 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#C#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:54:56,431 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:56,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:56,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:56,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:56,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:56,434 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:56,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:56,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:56,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,438 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:56,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:56,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c8246b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:56,443 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:56,443 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,445 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:56,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:56,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:56,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:56,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,447 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:56,449 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:56,449 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:56,449 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:56,449 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:56,449 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:56,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,451 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,452 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:56,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:56,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,454 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:56,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,454 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,454 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:56,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560556513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560556518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,555 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:56,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:56,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:56,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:56,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:56,558 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:56,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:56,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:56,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:56,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:56,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:56,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ce61b4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:56,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:56,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:56,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,571 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:56,571 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:56,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:56,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:56,574 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:56,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:56,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:56,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:56,574 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:56,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,580 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,582 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:56,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:56,583 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:56,583 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:56,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560556665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560556666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560556672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,683 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:56,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:56,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:56,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:56,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:56,686 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:56,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:56,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:56,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,686 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,687 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:56,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:56,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:56,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:56,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:56,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:56,689 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,689 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:56,690 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:56,690 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:56,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:56,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289170f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:56,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:56,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,697 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:56,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:56,700 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:56,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:56,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:56,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:56,700 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:56,702 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,702 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,702 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,702 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,705 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:56,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:56,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,706 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:56,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:56,706 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:56,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560556717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:56,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560556720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/79c15d7441624ef8be4f07850d51fc96 2023-04-27 01:54:56,799 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/2935a9470cc743589e45f6475a499a05 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2935a9470cc743589e45f6475a499a05 2023-04-27 01:54:56,806 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:56,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:56,807 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/B of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 2935a9470cc743589e45f6475a499a05(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:56,807 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:56,807 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/B, priority=10, startTime=1682560496339; duration=0sec 2023-04-27 01:54:56,807 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:56,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:56,810 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:56,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:56,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:56,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:56,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:56,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:56,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:56,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:56,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:56,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:56,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:56,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:56,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:56,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63a1c0ba[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:56,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:56,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,827 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:56,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/df42961a90d5496dbae5e1f381ed0e64 2023-04-27 01:54:56,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:56,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:56,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:56,834 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:56,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:56,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:56,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:56,834 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:56,835 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/17e0b7d13b5942e0860a80daa362dd64 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/17e0b7d13b5942e0860a80daa362dd64 2023-04-27 01:54:56,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:56,838 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,843 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:56,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:56,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:56,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:56,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:56,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,843 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,844 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:56,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:56,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,844 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,844 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:56,844 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,845 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/C of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 17e0b7d13b5942e0860a80daa362dd64(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:54:56,845 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:56,845 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/C, priority=10, startTime=1682560496339; duration=0sec 2023-04-27 01:54:56,845 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:54:56,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/b567be0f60e6424c9fe412e3f833e287 2023-04-27 01:54:56,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/79c15d7441624ef8be4f07850d51fc96 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/79c15d7441624ef8be4f07850d51fc96 2023-04-27 01:54:56,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/79c15d7441624ef8be4f07850d51fc96, entries=200, sequenceid=186, filesize=14.1 K 2023-04-27 01:54:56,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/df42961a90d5496dbae5e1f381ed0e64 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/df42961a90d5496dbae5e1f381ed0e64 2023-04-27 01:54:56,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/df42961a90d5496dbae5e1f381ed0e64, entries=150, sequenceid=186, filesize=11.8 K 2023-04-27 01:54:56,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/b567be0f60e6424c9fe412e3f833e287 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b567be0f60e6424c9fe412e3f833e287 2023-04-27 01:54:56,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b567be0f60e6424c9fe412e3f833e287, entries=150, sequenceid=186, filesize=11.8 K 2023-04-27 01:54:56,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~181.03 KB/185376, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 537ms, sequenceid=186, compaction requested=false 2023-04-27 01:54:56,886 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:56,944 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:56,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:56,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:56,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:56,947 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:56,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:56,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:56,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,948 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:56,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:56,949 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:56,949 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:56,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:56,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:56,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:56,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:56,950 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:56,950 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:56,951 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:56,951 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:56,951 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:56,951 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:56,951 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:56,951 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:56,951 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:57,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:57,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:57,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560557033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560557033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560557136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560557136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560557339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560557339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,360 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/6718f374431e42d1a3fd132a77297db5 2023-04-27 01:54:57,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560557641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:57,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560557643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,770 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/2a52887f4ab64fdf945deb704674c95b 2023-04-27 01:54:57,781 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/0c5f6fdd4d584cc1b5fac387439e7b88 2023-04-27 01:54:57,786 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/6718f374431e42d1a3fd132a77297db5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6718f374431e42d1a3fd132a77297db5 2023-04-27 01:54:57,789 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6718f374431e42d1a3fd132a77297db5, entries=150, sequenceid=209, filesize=11.8 K 2023-04-27 01:54:57,790 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/2a52887f4ab64fdf945deb704674c95b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2a52887f4ab64fdf945deb704674c95b 2023-04-27 01:54:57,794 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2a52887f4ab64fdf945deb704674c95b, entries=150, sequenceid=209, filesize=11.8 K 2023-04-27 01:54:57,794 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/0c5f6fdd4d584cc1b5fac387439e7b88 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0c5f6fdd4d584cc1b5fac387439e7b88 2023-04-27 01:54:57,798 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0c5f6fdd4d584cc1b5fac387439e7b88, entries=150, sequenceid=209, filesize=11.8 K 2023-04-27 01:54:57,798 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 848ms, sequenceid=209, compaction requested=true 2023-04-27 01:54:57,798 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:57,798 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:57,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@230a4bd9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:57,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:57,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:57,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:57,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:57,810 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:57,810 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:57,812 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:57,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:57,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:57,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:57,812 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:57,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:57,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:57,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:57,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:57,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:57,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:57,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:57,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:57,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:57,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:57,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:57,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,822 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:57,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:57,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:57,822 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:57,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:57,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,922 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:57,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:57,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:57,925 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:57,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:57,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:57,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:57,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:57,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:57,927 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:57,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:57,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:57,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:57,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:57,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:57,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:57,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:57,929 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:57,929 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:57,929 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:54:57,929 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:57,929 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:57,929 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:57,930 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:57,930 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:57,930 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:57,942 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/02cd7bb49d884a30bb5b34652c5546ce 2023-04-27 01:54:58,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:58,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:58,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560558166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560558168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560558269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560558270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,359 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/977fb7de596445bd96e2c3f9541df09f 2023-04-27 01:54:58,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560558473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560558474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560558672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,674 DEBUG [Thread-1529] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:54:58,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560558676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,679 DEBUG [Thread-1527] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:54:58,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560558686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,689 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:54:58,773 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/43589d3653be469fb8180a2349196cb1 2023-04-27 01:54:58,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682560558776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,777 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/02cd7bb49d884a30bb5b34652c5546ce as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/02cd7bb49d884a30bb5b34652c5546ce 2023-04-27 01:54:58,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:58,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38660 deadline: 1682560558776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,782 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/02cd7bb49d884a30bb5b34652c5546ce, entries=150, sequenceid=221, filesize=11.8 K 2023-04-27 01:54:58,783 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/977fb7de596445bd96e2c3f9541df09f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/977fb7de596445bd96e2c3f9541df09f 2023-04-27 01:54:58,788 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/977fb7de596445bd96e2c3f9541df09f, entries=150, sequenceid=221, filesize=11.8 K 2023-04-27 01:54:58,790 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/43589d3653be469fb8180a2349196cb1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/43589d3653be469fb8180a2349196cb1 2023-04-27 01:54:58,795 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/43589d3653be469fb8180a2349196cb1, entries=150, sequenceid=221, filesize=11.8 K 2023-04-27 01:54:58,795 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 866ms, sequenceid=221, compaction requested=true 2023-04-27 01:54:58,795 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:58,795 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:58,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:58,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:58,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:58,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:58,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ff5dcd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:58,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:58,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,801 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:58,802 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,802 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:58,804 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:58,807 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:58,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:58,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:58,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:58,807 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:58,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:58,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:58,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:58,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:58,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:58,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:58,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:58,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:58,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:58,813 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:58,813 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:58,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:58,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:58,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:54:58,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:58,913 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:58,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:58,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:58,916 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:58,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:58,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:58,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:58,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,917 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:58,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:58,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:58,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:58,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:58,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:58,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:58,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:58,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:58,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:58,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:58,920 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:59,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:59,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:59,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560559308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560559308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,329 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/e88d51f083c7414592860a753493f620 2023-04-27 01:54:59,339 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/d89f4477e1924de2a451e068ba48a4a7 2023-04-27 01:54:59,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560559412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560559412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560559613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560559614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,749 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/843497704bbd4a24bfb29dd9b33c7d7e 2023-04-27 01:54:59,753 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/e88d51f083c7414592860a753493f620 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/e88d51f083c7414592860a753493f620 2023-04-27 01:54:59,757 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/e88d51f083c7414592860a753493f620, entries=150, sequenceid=241, filesize=11.8 K 2023-04-27 01:54:59,758 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/d89f4477e1924de2a451e068ba48a4a7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d89f4477e1924de2a451e068ba48a4a7 2023-04-27 01:54:59,768 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d89f4477e1924de2a451e068ba48a4a7, entries=150, sequenceid=241, filesize=11.8 K 2023-04-27 01:54:59,769 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/843497704bbd4a24bfb29dd9b33c7d7e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/843497704bbd4a24bfb29dd9b33c7d7e 2023-04-27 01:54:59,774 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/843497704bbd4a24bfb29dd9b33c7d7e, entries=150, sequenceid=241, filesize=11.8 K 2023-04-27 01:54:59,774 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 854ms, sequenceid=241, compaction requested=true 2023-04-27 01:54:59,775 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:54:59,775 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:54:59,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,777 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:59,777 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,777 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:59,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:54:59,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f841112[Count = 0] remaining members to acquire global barrier 2023-04-27 01:54:59,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:54:59,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,780 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,780 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:54:59,780 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:54:59,780 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:54:59,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:54:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:54:59,784 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:59,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:54:59,787 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:54:59,787 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:54:59,787 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:54:59,787 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:54:59,787 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:54:59,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:54:59,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:54:59,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:59,789 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:54:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:59,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:54:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:54:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:59,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:59,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:54:59,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,793 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:54:59,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:54:59,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:54:59,794 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:59,794 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:54:59,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:59,794 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:54:59,794 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:54:59,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:54:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:54:59,894 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:54:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:54:59,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:54:59,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:54:59,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:54:59,897 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:54:59,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:59,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:54:59,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:54:59,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:59,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:54:59,900 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:54:59,900 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,900 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:59,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:54:59,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:54:59,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:54:59,901 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:54:59,901 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:54:59,901 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:54:59,902 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:54:59,902 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:54:59,902 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:54:59,902 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:54:59,909 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/514616ad436347388518d31aa3cfe733 2023-04-27 01:54:59,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:54:59,918 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:54:59,919 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/f6e9ed9f7b5b4003b86efb52203a8fa7 2023-04-27 01:54:59,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560559948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:54:59,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:54:59,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560559949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560560053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560560053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560560255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560560255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,329 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/84d1fc3034344fcd98a11d71b94a6c0d 2023-04-27 01:55:00,333 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/514616ad436347388518d31aa3cfe733 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/514616ad436347388518d31aa3cfe733 2023-04-27 01:55:00,337 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/514616ad436347388518d31aa3cfe733, entries=150, sequenceid=254, filesize=11.8 K 2023-04-27 01:55:00,338 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/f6e9ed9f7b5b4003b86efb52203a8fa7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f6e9ed9f7b5b4003b86efb52203a8fa7 2023-04-27 01:55:00,343 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f6e9ed9f7b5b4003b86efb52203a8fa7, entries=150, sequenceid=254, filesize=11.8 K 2023-04-27 01:55:00,343 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/84d1fc3034344fcd98a11d71b94a6c0d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/84d1fc3034344fcd98a11d71b94a6c0d 2023-04-27 01:55:00,347 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/84d1fc3034344fcd98a11d71b94a6c0d, entries=150, sequenceid=254, filesize=11.8 K 2023-04-27 01:55:00,347 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 445ms, sequenceid=254, compaction requested=true 2023-04-27 01:55:00,347 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:00,347 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:00,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:00,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:00,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:00,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:00,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,349 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:00,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:00,350 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,350 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:00,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:00,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cdbe48[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:00,351 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:00,351 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,352 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:00,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:00,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:00,353 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:00,354 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:00,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:00,356 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:00,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:00,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:00,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:00,356 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:00,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:00,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,358 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:00,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,359 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:00,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:00,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:00,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:00,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:00,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:00,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:00,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:00,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:00,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:00,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,362 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:00,362 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:00,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:00,462 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:00,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:00,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:00,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:00,466 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:00,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:00,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:00,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:00,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:00,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,469 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:00,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:00,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:00,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:00,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:00,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:00,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:00,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:00,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:00,471 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:55:00,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:00,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:00,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:00,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:00,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:00,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:00,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:00,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:00,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560560574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560560576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560560677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560560679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,879 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/b387e549d87849cdb2cdf3d21ffbbcc8 2023-04-27 01:55:00,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560560881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:00,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560560884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,890 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/fd61c6bf3d524160adda21d1f4e28af5 2023-04-27 01:55:00,900 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/410bd96287db465fa9316c8b16cd6b68 2023-04-27 01:55:00,904 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/b387e549d87849cdb2cdf3d21ffbbcc8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b387e549d87849cdb2cdf3d21ffbbcc8 2023-04-27 01:55:00,908 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b387e549d87849cdb2cdf3d21ffbbcc8, entries=150, sequenceid=273, filesize=11.9 K 2023-04-27 01:55:00,909 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/fd61c6bf3d524160adda21d1f4e28af5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/fd61c6bf3d524160adda21d1f4e28af5 2023-04-27 01:55:00,912 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/fd61c6bf3d524160adda21d1f4e28af5, entries=150, sequenceid=273, filesize=11.9 K 2023-04-27 01:55:00,914 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/410bd96287db465fa9316c8b16cd6b68 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/410bd96287db465fa9316c8b16cd6b68 2023-04-27 01:55:00,917 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/410bd96287db465fa9316c8b16cd6b68, entries=150, sequenceid=273, filesize=11.9 K 2023-04-27 01:55:00,918 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 447ms, sequenceid=273, compaction requested=true 2023-04-27 01:55:00,918 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:00,918 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:00,920 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43686f4b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:00,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:00,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,924 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:00,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:00,926 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:00,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:00,928 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:00,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:00,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:00,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:00,928 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:00,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:00,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:00,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:00,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:00,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:00,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:00,934 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:00,934 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:00,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:00,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:00,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:00,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:01,034 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:01,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:01,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:01,037 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:01,037 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:01,037 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:01,037 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:01,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:01,039 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,039 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:01,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:01,040 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,040 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:01,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:01,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:01,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:01,041 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:01,041 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:01,042 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:01,050 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/b56bb71c6c704ae783435bf8d0ad05f4 2023-04-27 01:55:01,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:01,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:01,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560561221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560561221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560561326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560561326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,460 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/1ade6b1a85354a7d96d8f3185122b695 2023-04-27 01:55:01,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560561531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560561531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560561836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:01,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560561836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,871 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/64c390630342474b823216c7994d2374 2023-04-27 01:55:01,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/b56bb71c6c704ae783435bf8d0ad05f4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b56bb71c6c704ae783435bf8d0ad05f4 2023-04-27 01:55:01,879 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b56bb71c6c704ae783435bf8d0ad05f4, entries=150, sequenceid=286, filesize=11.9 K 2023-04-27 01:55:01,880 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/1ade6b1a85354a7d96d8f3185122b695 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/1ade6b1a85354a7d96d8f3185122b695 2023-04-27 01:55:01,884 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/1ade6b1a85354a7d96d8f3185122b695, entries=150, sequenceid=286, filesize=11.9 K 2023-04-27 01:55:01,885 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/64c390630342474b823216c7994d2374 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/64c390630342474b823216c7994d2374 2023-04-27 01:55:01,888 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/64c390630342474b823216c7994d2374, entries=150, sequenceid=286, filesize=11.9 K 2023-04-27 01:55:01,889 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 848ms, sequenceid=286, compaction requested=true 2023-04-27 01:55:01,889 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:01,889 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:01,889 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:01,889 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:01,889 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,889 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:01,889 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:01,891 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1653de[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:01,893 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:01,893 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,894 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,894 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:01,894 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:01,894 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:01,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:01,895 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:01,898 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:01,899 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:01,899 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:01,899 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:01,899 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:01,901 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,901 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:01,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,901 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:01,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:01,903 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:01,907 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:01,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:01,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:01,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:01,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:01,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:01,907 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:01,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:01,908 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:01,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:01,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:01,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:01,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:02,008 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:02,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:02,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:02,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:02,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:02,011 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:02,011 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:02,011 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:02,011 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:02,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:02,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:02,013 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:02,013 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:02,013 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,013 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:02,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:02,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:02,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:02,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:02,014 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:02,014 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:02,015 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:02,015 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:02,015 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:02,015 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:02,015 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:02,015 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:02,015 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:02,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:02,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:02,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560562356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560562360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,422 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/65a613d0cb824b758a5283552af99a1f 2023-04-27 01:55:02,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560562461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560562465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560562664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560562670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38656 deadline: 1682560562695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,700 DEBUG [Thread-1529] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:02,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38678 deadline: 1682560562700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,704 DEBUG [Thread-1527] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:02,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38672 deadline: 1682560562711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,715 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:02,833 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/e50178aeffe241ecbe1a86dbd32ff159 2023-04-27 01:55:02,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560562968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:02,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:02,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560562974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,243 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/58dfc421b90548269f142263e5f254a8 2023-04-27 01:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/65a613d0cb824b758a5283552af99a1f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/65a613d0cb824b758a5283552af99a1f 2023-04-27 01:55:03,251 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/65a613d0cb824b758a5283552af99a1f, entries=150, sequenceid=306, filesize=11.9 K 2023-04-27 01:55:03,252 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/e50178aeffe241ecbe1a86dbd32ff159 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/e50178aeffe241ecbe1a86dbd32ff159 2023-04-27 01:55:03,256 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/e50178aeffe241ecbe1a86dbd32ff159, entries=150, sequenceid=306, filesize=11.9 K 2023-04-27 01:55:03,257 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/58dfc421b90548269f142263e5f254a8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/58dfc421b90548269f142263e5f254a8 2023-04-27 01:55:03,260 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/58dfc421b90548269f142263e5f254a8, entries=150, sequenceid=306, filesize=11.9 K 2023-04-27 01:55:03,261 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 1246ms, sequenceid=306, compaction requested=true 2023-04-27 01:55:03,261 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:03,261 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:03,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:03,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:03,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:03,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:03,263 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,263 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:03,263 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,263 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:03,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:03,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e5cbcf[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:03,265 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:03,265 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,267 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:03,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:03,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:03,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:03,268 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:03,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:03,269 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:03,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:03,271 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:03,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:03,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:03,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:03,271 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:03,273 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,273 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,273 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:03,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:03,273 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:03,275 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,277 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:03,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:03,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:03,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:03,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:03,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,277 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:03,277 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:03,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:03,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:03,378 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:03,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:03,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:03,381 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:03,381 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:03,381 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:03,381 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:03,381 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,381 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:03,383 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:03,383 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,384 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:03,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:03,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:03,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:03,385 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:03,385 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:03,385 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:03,385 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:03,385 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:03,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:03,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:03,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:03,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:03,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:03,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:03,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:03,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560563512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:03,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560563514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:03,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560563617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:03,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560563618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,793 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/7f98d278178f4f19aee4fccb929325d6 2023-04-27 01:55:03,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:03,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560563822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:03,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:03,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560563822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560564126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560564128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,203 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/7a168c95cff44556be73fab260a6feed 2023-04-27 01:55:04,213 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/ea401f4756a84b51b67f26afeb9872ba 2023-04-27 01:55:04,216 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/7f98d278178f4f19aee4fccb929325d6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7f98d278178f4f19aee4fccb929325d6 2023-04-27 01:55:04,220 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7f98d278178f4f19aee4fccb929325d6, entries=150, sequenceid=318, filesize=11.9 K 2023-04-27 01:55:04,221 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/7a168c95cff44556be73fab260a6feed as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/7a168c95cff44556be73fab260a6feed 2023-04-27 01:55:04,224 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/7a168c95cff44556be73fab260a6feed, entries=150, sequenceid=318, filesize=11.9 K 2023-04-27 01:55:04,225 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/ea401f4756a84b51b67f26afeb9872ba as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/ea401f4756a84b51b67f26afeb9872ba 2023-04-27 01:55:04,228 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/ea401f4756a84b51b67f26afeb9872ba, entries=150, sequenceid=318, filesize=11.9 K 2023-04-27 01:55:04,229 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 844ms, sequenceid=318, compaction requested=true 2023-04-27 01:55:04,229 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:04,229 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:04,229 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:04,229 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:04,229 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,230 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:04,230 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,231 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:04,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:04,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:04,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:04,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c850005[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:04,233 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:04,233 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,234 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:04,234 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:04,234 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:04,234 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,234 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:04,235 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:04,235 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:04,235 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:04,236 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:04,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:04,238 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:04,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:04,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:04,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:04,238 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:04,241 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,241 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:04,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,241 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:04,242 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:04,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:04,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:04,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,245 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:04,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:04,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:04,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:04,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:04,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:04,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:04,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,246 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:04,246 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:04,246 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:04,246 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:04,346 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:04,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:04,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:04,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:04,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:04,349 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:04,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:04,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:04,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:04,350 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,350 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:04,351 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:04,351 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:04,351 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,351 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:04,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:04,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:04,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:04,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:04,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:04,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:04,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:04,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:04,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:04,352 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:04,352 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:04,353 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:04,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:04,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:04,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:04,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:04,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:04,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:04,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:04,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:04,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560564651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560564652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560564755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560564757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,761 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/b442182a4e3b4121aa533f1e53a30f91 2023-04-27 01:55:04,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560564959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:04,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:04,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560564962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,174 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/cd875da1c39241e99db3d312cb907f5a 2023-04-27 01:55:05,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:05,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560565264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:05,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560565266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,584 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/dceaf8a0a0b745a3a5087c53f52dbda3 2023-04-27 01:55:05,589 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/b442182a4e3b4121aa533f1e53a30f91 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b442182a4e3b4121aa533f1e53a30f91 2023-04-27 01:55:05,593 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b442182a4e3b4121aa533f1e53a30f91, entries=150, sequenceid=338, filesize=11.9 K 2023-04-27 01:55:05,594 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/cd875da1c39241e99db3d312cb907f5a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/cd875da1c39241e99db3d312cb907f5a 2023-04-27 01:55:05,598 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/cd875da1c39241e99db3d312cb907f5a, entries=150, sequenceid=338, filesize=11.9 K 2023-04-27 01:55:05,599 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/dceaf8a0a0b745a3a5087c53f52dbda3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/dceaf8a0a0b745a3a5087c53f52dbda3 2023-04-27 01:55:05,602 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/dceaf8a0a0b745a3a5087c53f52dbda3, entries=150, sequenceid=338, filesize=11.9 K 2023-04-27 01:55:05,603 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 1250ms, sequenceid=338, compaction requested=true 2023-04-27 01:55:05,603 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:05,603 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:05,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:05,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:05,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:05,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:05,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c1724e9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:05,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:05,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,609 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:05,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:05,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:05,610 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,610 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:05,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:05,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:05,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:05,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:05,613 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:05,613 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:05,613 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:05,613 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:05,613 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:05,615 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,615 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:05,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,615 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:05,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,617 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:05,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:05,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:05,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:05,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:05,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:05,623 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:05,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,623 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:05,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:05,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:05,623 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:05,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,624 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,624 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,724 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:05,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:05,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:05,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:05,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:05,727 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:05,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:05,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:05,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:05,728 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,728 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,729 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:05,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:05,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:05,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:05,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:05,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:05,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:05,730 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:05,730 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:05,730 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:05,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:05,731 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:05,731 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:05,741 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/0023b903a707406f97b4a9c4e82a5c6d 2023-04-27 01:55:05,751 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/84618bd01d8543b1aede83656bb43023 2023-04-27 01:55:05,760 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/65ea97dcf2314fd5a1ae12e2d2e392f2 2023-04-27 01:55:05,767 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/0023b903a707406f97b4a9c4e82a5c6d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/0023b903a707406f97b4a9c4e82a5c6d 2023-04-27 01:55:05,772 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/0023b903a707406f97b4a9c4e82a5c6d, entries=150, sequenceid=350, filesize=11.9 K 2023-04-27 01:55:05,773 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/84618bd01d8543b1aede83656bb43023 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/84618bd01d8543b1aede83656bb43023 2023-04-27 01:55:05,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:05,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:05,779 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/84618bd01d8543b1aede83656bb43023, entries=150, sequenceid=350, filesize=11.9 K 2023-04-27 01:55:05,780 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/65ea97dcf2314fd5a1ae12e2d2e392f2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/65ea97dcf2314fd5a1ae12e2d2e392f2 2023-04-27 01:55:05,798 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/65ea97dcf2314fd5a1ae12e2d2e392f2, entries=150, sequenceid=350, filesize=11.9 K 2023-04-27 01:55:05,799 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=53.67 KB/54960 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 68ms, sequenceid=350, compaction requested=true 2023-04-27 01:55:05,799 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:05,799 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:05,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:05,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:05,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:05,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:05,801 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:05,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:05,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:05,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:05,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:05,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12811e16[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:05,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,805 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:05,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:05,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:05,805 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,806 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:05,807 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:05,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,807 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,807 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:05,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:05,810 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:05,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:05,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:05,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:05,810 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:05,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:05,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:05,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:05,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:05,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:05,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:05,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,818 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:05,818 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:05,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:05,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:05,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560565857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:05,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560565861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,918 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:05,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:05,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:05,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:05,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:05,921 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:05,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:05,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:05,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:05,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:05,929 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:05,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:05,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:05,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:05,931 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:05,931 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:05,931 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:05,931 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:05,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:05,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:05,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f47043a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:05,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:05,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,935 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:05,936 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:05,936 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:05,936 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:05,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:05,939 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:05,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:05,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:05,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:05,939 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:05,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:05,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:05,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:05,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:05,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:05,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:05,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:05,954 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:05,954 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:05,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:05,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:05,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:05,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:05,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:05,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:05,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:05,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:05,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:05,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:05,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560565963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:05,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:05,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560565966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,054 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,058 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,060 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:06,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,063 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,063 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,063 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,063 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,065 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be6b4ff[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,069 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,069 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,069 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,073 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,073 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,073 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,073 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,073 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,075 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,075 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,075 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,075 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,079 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:06,079 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:06,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560566170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:06,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560566175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,179 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,183 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:06,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,192 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,192 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,192 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,192 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,197 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee3f106[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,200 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,201 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,202 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,204 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,204 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,207 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,208 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,210 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:06,210 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,210 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,210 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/c93247e523b24625bf1dac2c48a64efa 2023-04-27 01:55:06,311 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,314 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,316 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,318 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,318 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,318 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,318 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64083162[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,324 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,324 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,324 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,324 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,324 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,325 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,325 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,328 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,328 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,328 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,328 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,328 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,330 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,331 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,333 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,333 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,334 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,434 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,437 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:06,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,441 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,442 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,442 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,442 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,442 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,442 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,442 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,442 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,442 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b98f0d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,446 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,446 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,447 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,447 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,447 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,450 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,450 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,450 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,450 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,450 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,452 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,452 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,452 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,456 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,456 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,456 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,457 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,457 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,457 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,457 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:06,457 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,457 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,457 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,457 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,457 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:06,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560566473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:06,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560566480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,557 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,561 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,562 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,562 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:06,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,564 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,564 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,564 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,564 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0694c9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,571 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,571 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,576 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,576 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,576 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,576 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,576 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,583 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,583 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,584 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,584 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,584 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,584 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,584 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/907df09a6cf448c2a39e146b5f75b3e2 2023-04-27 01:55:06,684 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,687 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:06,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,692 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,692 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,692 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,692 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8c37ae[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,696 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,696 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,697 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,698 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,700 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,701 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,703 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,703 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,703 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,707 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,707 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,807 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,811 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:06,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,815 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,815 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,815 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,815 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92a807c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,825 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,825 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,827 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,827 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,827 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,831 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,831 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,831 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,931 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:06,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:06,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:06,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:06,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:06,935 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:06,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:06,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:06,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:06,942 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,942 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:06,942 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:06,942 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:06,942 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:06,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,945 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:06,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7694b7b3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:06,946 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:06,946 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,947 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,947 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:06,947 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:06,947 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:06,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:06,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:06,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:06,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:06,954 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:06,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:06,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:06,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:06,954 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:06,955 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,955 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:06,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,955 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:06,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,957 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:06,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:06,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:06,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:06,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:06,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:06,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:06,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:06,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:06,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:06,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:06,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:06,961 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:06,961 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:06,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:06,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560566982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:06,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:06,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560566984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,033 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/f0cb059d576d4cce9e8d2165d390e421 2023-04-27 01:55:07,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/c93247e523b24625bf1dac2c48a64efa as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c93247e523b24625bf1dac2c48a64efa 2023-04-27 01:55:07,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c93247e523b24625bf1dac2c48a64efa, entries=200, sequenceid=362, filesize=14.3 K 2023-04-27 01:55:07,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/907df09a6cf448c2a39e146b5f75b3e2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/907df09a6cf448c2a39e146b5f75b3e2 2023-04-27 01:55:07,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/907df09a6cf448c2a39e146b5f75b3e2, entries=150, sequenceid=362, filesize=11.9 K 2023-04-27 01:55:07,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/f0cb059d576d4cce9e8d2165d390e421 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f0cb059d576d4cce9e8d2165d390e421 2023-04-27 01:55:07,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f0cb059d576d4cce9e8d2165d390e421, entries=150, sequenceid=362, filesize=11.9 K 2023-04-27 01:55:07,059 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 1257ms, sequenceid=362, compaction requested=true 2023-04-27 01:55:07,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:07,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:07,059 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 13 store files, 0 compacting, 13 eligible, 16 blocking 2023-04-27 01:55:07,059 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 13 store files, 0 compacting, 13 eligible, 16 blocking 2023-04-27 01:55:07,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:07,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:07,061 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:07,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 121260 starting at candidate #2 after considering 60 permutations with 60 in ratio 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/A is initiating minor compaction 2023-04-27 01:55:07,064 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/A in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:07,064 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6718f374431e42d1a3fd132a77297db5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/02cd7bb49d884a30bb5b34652c5546ce, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/e88d51f083c7414592860a753493f620, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/514616ad436347388518d31aa3cfe733, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b387e549d87849cdb2cdf3d21ffbbcc8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b56bb71c6c704ae783435bf8d0ad05f4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/65a613d0cb824b758a5283552af99a1f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7f98d278178f4f19aee4fccb929325d6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b442182a4e3b4121aa533f1e53a30f91, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/0023b903a707406f97b4a9c4e82a5c6d] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=118.4 K 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 6718f374431e42d1a3fd132a77297db5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 02cd7bb49d884a30bb5b34652c5546ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e88d51f083c7414592860a753493f620, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 514616ad436347388518d31aa3cfe733, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b387e549d87849cdb2cdf3d21ffbbcc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b56bb71c6c704ae783435bf8d0ad05f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 65a613d0cb824b758a5283552af99a1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 7f98d278178f4f19aee4fccb929325d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b442182a4e3b4121aa533f1e53a30f91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338 2023-04-27 01:55:07,064 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 0023b903a707406f97b4a9c4e82a5c6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 121110 starting at candidate #1 after considering 60 permutations with 60 in ratio 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/B is initiating minor compaction 2023-04-27 01:55:07,067 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/B in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:07,067 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/df42961a90d5496dbae5e1f381ed0e64, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2a52887f4ab64fdf945deb704674c95b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/977fb7de596445bd96e2c3f9541df09f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d89f4477e1924de2a451e068ba48a4a7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f6e9ed9f7b5b4003b86efb52203a8fa7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/fd61c6bf3d524160adda21d1f4e28af5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/1ade6b1a85354a7d96d8f3185122b695, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/e50178aeffe241ecbe1a86dbd32ff159, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/7a168c95cff44556be73fab260a6feed, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/cd875da1c39241e99db3d312cb907f5a] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=118.3 K 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting df42961a90d5496dbae5e1f381ed0e64, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2a52887f4ab64fdf945deb704674c95b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 977fb7de596445bd96e2c3f9541df09f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d89f4477e1924de2a451e068ba48a4a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f6e9ed9f7b5b4003b86efb52203a8fa7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254 2023-04-27 01:55:07,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting fd61c6bf3d524160adda21d1f4e28af5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273 2023-04-27 01:55:07,068 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1ade6b1a85354a7d96d8f3185122b695, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286 2023-04-27 01:55:07,068 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e50178aeffe241ecbe1a86dbd32ff159, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306 2023-04-27 01:55:07,068 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7a168c95cff44556be73fab260a6feed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318 2023-04-27 01:55:07,068 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting cd875da1c39241e99db3d312cb907f5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338 2023-04-27 01:55:07,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:07,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:07,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:07,086 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:07,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:07,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:07,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:07,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:07,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:07,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,091 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:07,091 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#A#compaction#261 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:07,094 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#B#compaction#262 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:07,095 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:07,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:07,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:07,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:07,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:07,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:07,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:07,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:07,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:07,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:07,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:07,100 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:07,100 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:07,100 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:07,101 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:07,101 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:07,101 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:07,101 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:07,101 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:07,101 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:07,101 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:07,117 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/de8c1d62cf6f4620a2194fd621c73053 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/de8c1d62cf6f4620a2194fd621c73053 2023-04-27 01:55:07,122 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/A of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into de8c1d62cf6f4620a2194fd621c73053(size=12.2 K), total size for store is 52.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:07,122 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:07,122 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/A, priority=3, startTime=1682560507059; duration=0sec 2023-04-27 01:55:07,122 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:07,122 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 13 store files, 0 compacting, 13 eligible, 16 blocking 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 121110 starting at candidate #1 after considering 60 permutations with 60 in ratio 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/C is initiating minor compaction 2023-04-27 01:55:07,127 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/C in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:07,127 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b567be0f60e6424c9fe412e3f833e287, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0c5f6fdd4d584cc1b5fac387439e7b88, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/43589d3653be469fb8180a2349196cb1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/843497704bbd4a24bfb29dd9b33c7d7e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/84d1fc3034344fcd98a11d71b94a6c0d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/410bd96287db465fa9316c8b16cd6b68, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/64c390630342474b823216c7994d2374, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/58dfc421b90548269f142263e5f254a8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/ea401f4756a84b51b67f26afeb9872ba, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/dceaf8a0a0b745a3a5087c53f52dbda3] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=118.3 K 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b567be0f60e6424c9fe412e3f833e287, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 0c5f6fdd4d584cc1b5fac387439e7b88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 43589d3653be469fb8180a2349196cb1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 843497704bbd4a24bfb29dd9b33c7d7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 84d1fc3034344fcd98a11d71b94a6c0d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 410bd96287db465fa9316c8b16cd6b68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 64c390630342474b823216c7994d2374, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286 2023-04-27 01:55:07,127 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 58dfc421b90548269f142263e5f254a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306 2023-04-27 01:55:07,128 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting ea401f4756a84b51b67f26afeb9872ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318 2023-04-27 01:55:07,128 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting dceaf8a0a0b745a3a5087c53f52dbda3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338 2023-04-27 01:55:07,133 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/9fa8607404d741b2933ba38df5ce5317 2023-04-27 01:55:07,151 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#C#compaction#265 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:07,171 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/b53bfd2efffd46f0a3706279845775b7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b53bfd2efffd46f0a3706279845775b7 2023-04-27 01:55:07,180 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/C of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into b53bfd2efffd46f0a3706279845775b7(size=12.2 K), total size for store is 48.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:07,181 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:07,181 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/C, priority=3, startTime=1682560507059; duration=0sec 2023-04-27 01:55:07,181 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:07,510 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/403ba414b4e6460baf8e51c95cd6c410 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/403ba414b4e6460baf8e51c95cd6c410 2023-04-27 01:55:07,515 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/B of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 403ba414b4e6460baf8e51c95cd6c410(size=12.2 K), total size for store is 48.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:07,515 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:07,515 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/B, priority=3, startTime=1682560507059; duration=0sec 2023-04-27 01:55:07,515 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:07,562 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/8428d978dc8d4dd294baaa4e41bb6665 2023-04-27 01:55:07,973 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/a3b84539dfa047b68694134b2af88343 2023-04-27 01:55:07,977 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/9fa8607404d741b2933ba38df5ce5317 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9fa8607404d741b2933ba38df5ce5317 2023-04-27 01:55:07,981 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9fa8607404d741b2933ba38df5ce5317, entries=150, sequenceid=382, filesize=11.9 K 2023-04-27 01:55:07,982 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/8428d978dc8d4dd294baaa4e41bb6665 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8428d978dc8d4dd294baaa4e41bb6665 2023-04-27 01:55:07,985 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8428d978dc8d4dd294baaa4e41bb6665, entries=150, sequenceid=382, filesize=11.9 K 2023-04-27 01:55:07,986 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/a3b84539dfa047b68694134b2af88343 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a3b84539dfa047b68694134b2af88343 2023-04-27 01:55:07,989 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a3b84539dfa047b68694134b2af88343, entries=150, sequenceid=382, filesize=11.9 K 2023-04-27 01:55:07,990 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 890ms, sequenceid=382, compaction requested=true 2023-04-27 01:55:07,990 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:07,990 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:07,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:07,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:07,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:07,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:07,991 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:07,991 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:07,992 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:07,992 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:07,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:07,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f4dd574[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:07,993 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:07,993 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:07,994 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:07,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:07,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:07,994 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:07,994 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:07,994 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:07,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:07,996 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:07,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:07,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:07,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:07,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:07,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:07,998 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:07,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:07,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:07,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:07,998 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:08,001 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,001 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,001 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,001 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:08,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:08,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:08,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:08,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:08,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:08,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:08,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:08,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:08,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:08,005 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:08,005 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:08,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:08,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560508025, max timestamp=1682560508036Num uniques 859; 2023-04-27 01:55:08,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:08,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560508025, max timestamp=1682560508036Num uniques 859; 2023-04-27 01:55:08,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560508025, max timestamp=1682560508036Num uniques 859; 2023-04-27 01:55:08,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:08,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:08,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560568094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560568094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,105 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:08,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:08,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:08,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:08,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:08,109 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:08,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:08,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:08,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,110 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,111 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,111 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:08,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:08,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:08,112 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:08,112 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,112 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:08,112 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:08,112 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:08,112 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,112 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:08,112 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:08,113 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:08,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:08,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:08,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:08,114 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,115 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,115 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42b1b274[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:08,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:08,117 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,121 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,121 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:08,121 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:08,121 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:08,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:08,122 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:08,122 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,122 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:08,122 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:08,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:08,126 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:08,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:08,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:08,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:08,126 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:08,127 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,127 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,127 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,129 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,131 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:08,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:08,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,131 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:08,131 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:08,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560568198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560568198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,232 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:08,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:08,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:08,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:08,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:08,235 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:08,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:08,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:08,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,236 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,238 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:08,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:08,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,239 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:08,240 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:08,240 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,240 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:08,240 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:08,240 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,240 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:08,240 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:08,241 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,241 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,242 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,242 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f45f33e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:08,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:08,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,244 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:08,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:08,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:08,244 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,244 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:08,245 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:08,245 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,246 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:08,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:08,248 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:08,248 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:08,248 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:08,248 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:08,248 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:08,250 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,250 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,251 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,251 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,252 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,257 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:08,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:08,257 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:08,257 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:08,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,358 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:08,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:08,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:08,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:08,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:08,361 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:08,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:08,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:08,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,362 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,363 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,363 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,363 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,363 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:08,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:08,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:08,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:08,364 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,365 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:08,365 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:08,365 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:08,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:08,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:08,365 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:08,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:08,366 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:08,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3fe64a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:08,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:08,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,369 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,369 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:08,369 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:08,369 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:08,370 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,370 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:08,372 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:08,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:08,374 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:08,375 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:08,375 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:08,375 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:08,375 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:08,376 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,376 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,376 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,378 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:08,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:08,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,381 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,380 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,381 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:08,381 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:08,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:08,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560568401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560568402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/919705293ec94433a5cee7bbed81e3a7 2023-04-27 01:55:08,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/f44698c928984c0c882dbef277c6b021 2023-04-27 01:55:08,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/44ca530c271f4fea8a98cb770d6250d6 2023-04-27 01:55:08,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/919705293ec94433a5cee7bbed81e3a7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/919705293ec94433a5cee7bbed81e3a7 2023-04-27 01:55:08,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/919705293ec94433a5cee7bbed81e3a7, entries=150, sequenceid=395, filesize=11.9 K 2023-04-27 01:55:08,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/f44698c928984c0c882dbef277c6b021 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f44698c928984c0c882dbef277c6b021 2023-04-27 01:55:08,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f44698c928984c0c882dbef277c6b021, entries=150, sequenceid=395, filesize=11.9 K 2023-04-27 01:55:08,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/44ca530c271f4fea8a98cb770d6250d6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/44ca530c271f4fea8a98cb770d6250d6 2023-04-27 01:55:08,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/44ca530c271f4fea8a98cb770d6250d6, entries=150, sequenceid=395, filesize=11.9 K 2023-04-27 01:55:08,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 456ms, sequenceid=395, compaction requested=true 2023-04-27 01:55:08,459 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:08,459 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:08,459 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:08,459 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:08,459 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:08,459 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:08,461 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 78330 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:08,461 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/A is initiating minor compaction (all files) 2023-04-27 01:55:08,461 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/A in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,461 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/fa9d04a68fce48cca8e853332837e4a4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/79c15d7441624ef8be4f07850d51fc96, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/de8c1d62cf6f4620a2194fd621c73053, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c93247e523b24625bf1dac2c48a64efa, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9fa8607404d741b2933ba38df5ce5317, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/919705293ec94433a5cee7bbed81e3a7] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=76.5 K 2023-04-27 01:55:08,462 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting fa9d04a68fce48cca8e853332837e4a4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682560495152 2023-04-27 01:55:08,462 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73650 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:08,462 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/B is initiating minor compaction (all files) 2023-04-27 01:55:08,462 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/B in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,462 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2935a9470cc743589e45f6475a499a05, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/403ba414b4e6460baf8e51c95cd6c410, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/84618bd01d8543b1aede83656bb43023, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/907df09a6cf448c2a39e146b5f75b3e2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8428d978dc8d4dd294baaa4e41bb6665, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f44698c928984c0c882dbef277c6b021] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=71.9 K 2023-04-27 01:55:08,462 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 79c15d7441624ef8be4f07850d51fc96, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682560496324 2023-04-27 01:55:08,463 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2935a9470cc743589e45f6475a499a05, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682560495152 2023-04-27 01:55:08,463 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting de8c1d62cf6f4620a2194fd621c73053, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682560504645 2023-04-27 01:55:08,463 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 403ba414b4e6460baf8e51c95cd6c410, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682560503509 2023-04-27 01:55:08,463 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c93247e523b24625bf1dac2c48a64efa, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560505791 2023-04-27 01:55:08,464 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 84618bd01d8543b1aede83656bb43023, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682560504645 2023-04-27 01:55:08,464 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9fa8607404d741b2933ba38df5ce5317, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1682560505851 2023-04-27 01:55:08,464 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 907df09a6cf448c2a39e146b5f75b3e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560505795 2023-04-27 01:55:08,464 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 919705293ec94433a5cee7bbed81e3a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682560507992 2023-04-27 01:55:08,465 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8428d978dc8d4dd294baaa4e41bb6665, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1682560505851 2023-04-27 01:55:08,465 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f44698c928984c0c882dbef277c6b021, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682560507992 2023-04-27 01:55:08,481 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:08,482 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#B#compaction#271 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:08,483 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#A#compaction#270 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:08,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:08,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:08,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:08,488 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:08,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:08,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:08,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,488 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,489 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:08,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:08,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:08,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:08,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:08,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:08,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:08,502 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:08,504 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,504 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:08,504 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-27 01:55:08,504 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:08,504 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:08,505 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:08,505 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:08,505 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:08,505 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:08,515 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/d0b30c0ad19e49d4a3df50bd3f194fef as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/d0b30c0ad19e49d4a3df50bd3f194fef 2023-04-27 01:55:08,520 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/A of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into d0b30c0ad19e49d4a3df50bd3f194fef(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:08,520 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:08,520 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/A, priority=10, startTime=1682560508459; duration=0sec 2023-04-27 01:55:08,520 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:08,520 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:08,525 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73650 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:08,525 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/C is initiating minor compaction (all files) 2023-04-27 01:55:08,525 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/C in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,525 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/17e0b7d13b5942e0860a80daa362dd64, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b53bfd2efffd46f0a3706279845775b7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/65ea97dcf2314fd5a1ae12e2d2e392f2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f0cb059d576d4cce9e8d2165d390e421, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a3b84539dfa047b68694134b2af88343, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/44ca530c271f4fea8a98cb770d6250d6] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=71.9 K 2023-04-27 01:55:08,525 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 17e0b7d13b5942e0860a80daa362dd64, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1682560495152 2023-04-27 01:55:08,526 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b53bfd2efffd46f0a3706279845775b7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1682560503509 2023-04-27 01:55:08,526 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 65ea97dcf2314fd5a1ae12e2d2e392f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682560504645 2023-04-27 01:55:08,527 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f0cb059d576d4cce9e8d2165d390e421, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560505795 2023-04-27 01:55:08,527 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a3b84539dfa047b68694134b2af88343, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1682560505851 2023-04-27 01:55:08,528 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 44ca530c271f4fea8a98cb770d6250d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682560507992 2023-04-27 01:55:08,552 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#C#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:08,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:08,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:08,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560568716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560568717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560568821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560568821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,902 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/04d520ca08c443e393b23cbfaeb1e0d2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04d520ca08c443e393b23cbfaeb1e0d2 2023-04-27 01:55:08,907 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/B of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 04d520ca08c443e393b23cbfaeb1e0d2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:08,907 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:08,907 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/B, priority=10, startTime=1682560508459; duration=0sec 2023-04-27 01:55:08,907 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:08,928 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/eac35a97b9ab4472a3f9da07352db732 2023-04-27 01:55:08,938 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/04ec62dd033e4eabbfeb408769ede386 2023-04-27 01:55:08,947 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/c4217af7e2a54218b3a02824097800ea 2023-04-27 01:55:08,951 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/eac35a97b9ab4472a3f9da07352db732 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/eac35a97b9ab4472a3f9da07352db732 2023-04-27 01:55:08,954 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/eac35a97b9ab4472a3f9da07352db732, entries=150, sequenceid=432, filesize=11.9 K 2023-04-27 01:55:08,955 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/04ec62dd033e4eabbfeb408769ede386 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04ec62dd033e4eabbfeb408769ede386 2023-04-27 01:55:08,958 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04ec62dd033e4eabbfeb408769ede386, entries=150, sequenceid=432, filesize=11.9 K 2023-04-27 01:55:08,959 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/c4217af7e2a54218b3a02824097800ea as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c4217af7e2a54218b3a02824097800ea 2023-04-27 01:55:08,962 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c4217af7e2a54218b3a02824097800ea, entries=150, sequenceid=432, filesize=11.9 K 2023-04-27 01:55:08,963 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 459ms, sequenceid=432, compaction requested=false 2023-04-27 01:55:08,963 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:08,963 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:08,963 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:08,963 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:08,963 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:08,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:08,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:08,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d386ccc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:08,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:08,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,969 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,969 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:08,969 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:08,969 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:08,969 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,970 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:08,971 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:08,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,972 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/8677e24caa1e487cb6062d527c321371 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/8677e24caa1e487cb6062d527c321371 2023-04-27 01:55:08,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:08,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:08,974 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:08,974 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:08,974 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:08,974 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:08,974 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:08,976 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,976 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,977 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:08,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:08,977 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/C of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 8677e24caa1e487cb6062d527c321371(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:08,977 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:08,977 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/C, priority=10, startTime=1682560508459; duration=0sec 2023-04-27 01:55:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,977 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:08,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:08,979 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:08,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,981 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,984 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:08,984 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:08,984 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:08,984 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:08,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:08,984 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,985 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:08,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:08,985 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:08,985 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,985 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:08,985 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:08,985 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:08,985 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:08,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:09,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:09,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:09,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:09,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:09,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:09,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:09,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:09,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560509046, max timestamp=1682560509052Num uniques 859; 2023-04-27 01:55:09,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560509046, max timestamp=1682560509052Num uniques 859; 2023-04-27 01:55:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560509046, max timestamp=1682560509052Num uniques 859; 2023-04-27 01:55:09,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:09,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560569080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560569081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,085 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,088 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,090 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,090 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,090 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,090 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:09,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:09,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:09,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,095 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,095 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:09,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d5590d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:09,096 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:09,096 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,097 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:09,098 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,098 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:09,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:09,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,099 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:09,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:09,102 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:09,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:09,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:09,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:09,102 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:09,103 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,103 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,103 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,104 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,105 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,110 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:09,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,110 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:09,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,111 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:09,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560569184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560569185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,211 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,214 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,214 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,214 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,214 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,214 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,214 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,216 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,216 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,217 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,217 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,218 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,218 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,218 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:09,218 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:09,220 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:09,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b99a6a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:09,222 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:09,222 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,223 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:09,223 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,223 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:09,224 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,224 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,224 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,225 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:09,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:09,227 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:09,227 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:09,227 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:09,227 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:09,227 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:09,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,236 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,238 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,238 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,238 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,238 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:09,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:09,239 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,239 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,239 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:09,239 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,239 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:09,239 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,239 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,239 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,339 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,342 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,344 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,344 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,345 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,345 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,346 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,346 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,347 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:09,347 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:09,348 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,348 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,349 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61872bc1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:09,350 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:09,350 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,351 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:09,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:09,352 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:09,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:09,353 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:09,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:09,356 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:09,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:09,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:09,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:09,356 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:09,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,358 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,360 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:09,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,362 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:09,362 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:09,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,363 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:09,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560569387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560569389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/6d3dd1f9edb646a29ad7799f0eeadafa 2023-04-27 01:55:09,463 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,466 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,469 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:09,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:09,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:09,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:09,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:09,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:09,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f377531[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:09,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:09,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,476 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:09,476 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,476 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:09,476 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:09,477 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,477 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:09,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:09,480 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:09,480 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:09,480 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:09,480 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:09,480 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:09,482 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,482 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,482 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,483 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,486 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:09,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:09,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,486 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,487 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:09,487 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:09,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,587 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,591 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,595 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:09,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:09,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,598 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:09,598 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:09,600 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:09,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476248eb[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:09,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:09,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:09,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:09,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:09,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:09,607 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:09,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:09,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:09,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:09,607 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:09,608 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,608 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,609 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,610 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:09,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:09,613 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,614 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:09,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,614 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:09,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560569689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:09,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560569692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,714 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,717 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,718 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,718 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,718 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,719 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,721 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,721 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,721 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:09,721 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:09,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:09,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5996f0a7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:09,724 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:09,724 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,725 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:09,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,728 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:09,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:09,730 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:09,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:09,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:09,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:09,730 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:09,732 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,732 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,732 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,732 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,733 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:09,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:09,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:09,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,736 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:09,736 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:09,735 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,736 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,736 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,836 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,839 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,842 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,842 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,843 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,843 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:09,843 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,843 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:09,843 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:09,843 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,843 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:09,843 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:09,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/387e533adadb4d4890d137be830b8878 2023-04-27 01:55:09,845 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,846 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,846 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1170177[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:09,847 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:09,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,849 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:09,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:09,849 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:09,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,852 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:09,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/9511d5155f3b4cc2bc8412a802cdc05b 2023-04-27 01:55:09,855 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:09,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:09,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:09,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:09,855 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:09,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:09,857 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,857 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,857 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:09,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,858 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:09,859 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/6d3dd1f9edb646a29ad7799f0eeadafa as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6d3dd1f9edb646a29ad7799f0eeadafa 2023-04-27 01:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,860 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6d3dd1f9edb646a29ad7799f0eeadafa, entries=150, sequenceid=445, filesize=11.9 K 2023-04-27 01:55:09,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/387e533adadb4d4890d137be830b8878 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/387e533adadb4d4890d137be830b8878 2023-04-27 01:55:09,864 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,864 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:09,864 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:09,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:09,864 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,864 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:09,864 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,865 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,865 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,865 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:09,865 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:09,865 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,865 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:09,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:09,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/387e533adadb4d4890d137be830b8878, entries=150, sequenceid=445, filesize=11.9 K 2023-04-27 01:55:09,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/9511d5155f3b4cc2bc8412a802cdc05b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/9511d5155f3b4cc2bc8412a802cdc05b 2023-04-27 01:55:09,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/9511d5155f3b4cc2bc8412a802cdc05b, entries=150, sequenceid=445, filesize=11.9 K 2023-04-27 01:55:09,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 851ms, sequenceid=445, compaction requested=true 2023-04-27 01:55:09,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:09,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:09,877 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:09,877 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:09,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:09,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:09,878 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:09,878 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/A is initiating minor compaction (all files) 2023-04-27 01:55:09,878 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:09,878 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/A in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,878 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/B is initiating minor compaction (all files) 2023-04-27 01:55:09,878 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/d0b30c0ad19e49d4a3df50bd3f194fef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/eac35a97b9ab4472a3f9da07352db732, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6d3dd1f9edb646a29ad7799f0eeadafa] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=36.6 K 2023-04-27 01:55:09,878 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/B in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,878 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04d520ca08c443e393b23cbfaeb1e0d2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04ec62dd033e4eabbfeb408769ede386, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/387e533adadb4d4890d137be830b8878] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=36.6 K 2023-04-27 01:55:09,879 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d0b30c0ad19e49d4a3df50bd3f194fef, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682560507992 2023-04-27 01:55:09,879 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 04d520ca08c443e393b23cbfaeb1e0d2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682560507992 2023-04-27 01:55:09,879 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting eac35a97b9ab4472a3f9da07352db732, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682560508064 2023-04-27 01:55:09,879 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 04ec62dd033e4eabbfeb408769ede386, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682560508064 2023-04-27 01:55:09,879 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 6d3dd1f9edb646a29ad7799f0eeadafa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682560508711 2023-04-27 01:55:09,879 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 387e533adadb4d4890d137be830b8878, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682560508711 2023-04-27 01:55:09,885 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#B#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:09,885 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#A#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:09,911 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/5b64b744fd274475b08c7aa77daf7fd4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/5b64b744fd274475b08c7aa77daf7fd4 2023-04-27 01:55:09,915 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/B of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 5b64b744fd274475b08c7aa77daf7fd4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:09,915 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:09,915 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/B, priority=13, startTime=1682560509877; duration=0sec 2023-04-27 01:55:09,915 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:09,915 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:09,916 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:09,916 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): 6f2bfc2234dd9d4cdcd80dce1fbcb604/C is initiating minor compaction (all files) 2023-04-27 01:55:09,916 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f2bfc2234dd9d4cdcd80dce1fbcb604/C in TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,916 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/8677e24caa1e487cb6062d527c321371, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c4217af7e2a54218b3a02824097800ea, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/9511d5155f3b4cc2bc8412a802cdc05b] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp, totalSize=36.6 K 2023-04-27 01:55:09,916 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8677e24caa1e487cb6062d527c321371, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682560507992 2023-04-27 01:55:09,917 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c4217af7e2a54218b3a02824097800ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1682560508064 2023-04-27 01:55:09,918 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9511d5155f3b4cc2bc8412a802cdc05b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682560508711 2023-04-27 01:55:09,923 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f2bfc2234dd9d4cdcd80dce1fbcb604#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:09,943 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/6173151927ff48139d413b46143046c1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/6173151927ff48139d413b46143046c1 2023-04-27 01:55:09,947 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/C of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into 6173151927ff48139d413b46143046c1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:09,947 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:09,947 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/C, priority=13, startTime=1682560509877; duration=0sec 2023-04-27 01:55:09,947 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:09,965 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:09,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:09,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:09,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:09,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:09,968 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:09,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:09,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:09,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,969 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,969 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:09,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:09,970 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:09,970 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:09,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:09,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:09,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:09,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:09,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:09,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:09,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:09,971 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:09,971 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:09,971 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:09,971 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 01:55:09,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:09,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:09,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:09,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:09,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:09,972 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:10,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:10,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:10,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560570205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560570208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560570309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560570311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,313 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/f7b26da067084d28b3284a4c105c7b45 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/f7b26da067084d28b3284a4c105c7b45 2023-04-27 01:55:10,317 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6f2bfc2234dd9d4cdcd80dce1fbcb604/A of 6f2bfc2234dd9d4cdcd80dce1fbcb604 into f7b26da067084d28b3284a4c105c7b45(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:10,317 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:10,317 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604., storeName=6f2bfc2234dd9d4cdcd80dce1fbcb604/A, priority=13, startTime=1682560509877; duration=0sec 2023-04-27 01:55:10,317 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:10,385 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/4492735c3d0145ff8749e741dff09604 2023-04-27 01:55:10,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560570513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560570513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,796 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/67663244c16a48b2a8ea849697f7bda1 2023-04-27 01:55:10,806 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/1888176f8ac346f292dfcc48fc2bd7db 2023-04-27 01:55:10,810 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/4492735c3d0145ff8749e741dff09604 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/4492735c3d0145ff8749e741dff09604 2023-04-27 01:55:10,813 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/4492735c3d0145ff8749e741dff09604, entries=150, sequenceid=483, filesize=11.9 K 2023-04-27 01:55:10,814 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/67663244c16a48b2a8ea849697f7bda1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/67663244c16a48b2a8ea849697f7bda1 2023-04-27 01:55:10,818 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/67663244c16a48b2a8ea849697f7bda1, entries=150, sequenceid=483, filesize=11.9 K 2023-04-27 01:55:10,818 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/1888176f8ac346f292dfcc48fc2bd7db as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/1888176f8ac346f292dfcc48fc2bd7db 2023-04-27 01:55:10,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38654 deadline: 1682560570815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:10,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38660 deadline: 1682560570817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,822 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/1888176f8ac346f292dfcc48fc2bd7db, entries=150, sequenceid=483, filesize=11.9 K 2023-04-27 01:55:10,822 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 851ms, sequenceid=483, compaction requested=false 2023-04-27 01:55:10,822 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:10,822 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:10,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:10,823 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:10,823 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,823 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:10,823 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:10,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:10,826 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,827 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:10,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:10,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35bd6299[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:10,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:10,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:10,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:10,830 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:10,830 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,830 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:10,831 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,831 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:10,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:10,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:10,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:10,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:10,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:10,834 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:10,834 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:10,835 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:10,835 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:10,835 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:10,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:10,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:10,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:10,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:10,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:10,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:10,838 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:10,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:10,840 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,840 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:10,840 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:10,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:10,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:10,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:10,840 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:10,840 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:10,840 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:10,840 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:10,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:10,841 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:10,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:10,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:10,841 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,941 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:10,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:10,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:10,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:10,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:10,944 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:10,944 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:10,944 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:10,945 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:10,945 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,945 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:10,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,947 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:10,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:10,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:10,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:10,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:10,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:10,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:10,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:10,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:10,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:10,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:10,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:10,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:10,949 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:10,949 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. started... 2023-04-27 01:55:10,949 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:10,949 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:11,150 DEBUG [Thread-1544] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681ace6c to 127.0.0.1:52731 2023-04-27 01:55:11,150 DEBUG [Thread-1542] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f29c1c4 to 127.0.0.1:52731 2023-04-27 01:55:11,151 DEBUG [Thread-1544] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:11,151 DEBUG [Thread-1542] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:11,151 DEBUG [Thread-1536] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b39d5b8 to 127.0.0.1:52731 2023-04-27 01:55:11,151 DEBUG [Thread-1536] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:11,151 DEBUG [Thread-1540] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e5b1c14 to 127.0.0.1:52731 2023-04-27 01:55:11,151 DEBUG [Thread-1540] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:11,152 DEBUG [Thread-1538] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44f9bd75 to 127.0.0.1:52731 2023-04-27 01:55:11,152 DEBUG [Thread-1538] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:11,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. as already flushing 2023-04-27 01:55:11,321 DEBUG [Thread-1531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b79c2c to 127.0.0.1:52731 2023-04-27 01:55:11,321 DEBUG [Thread-1531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:11,322 DEBUG [Thread-1533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5a81fa to 127.0.0.1:52731 2023-04-27 01:55:11,322 DEBUG [Thread-1533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:11,357 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/7c106f4d6e2b4515b4f354ac524dcc9a 2023-04-27 01:55:11,765 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/712a6d2afd734aa8a407fcd1d455400b 2023-04-27 01:55:12,175 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/0539201f82c64d8cb24f614f951336b9 2023-04-27 01:55:12,179 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/7c106f4d6e2b4515b4f354ac524dcc9a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7c106f4d6e2b4515b4f354ac524dcc9a 2023-04-27 01:55:12,183 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7c106f4d6e2b4515b4f354ac524dcc9a, entries=100, sequenceid=494, filesize=9.5 K 2023-04-27 01:55:12,183 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/712a6d2afd734aa8a407fcd1d455400b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/712a6d2afd734aa8a407fcd1d455400b 2023-04-27 01:55:12,187 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/712a6d2afd734aa8a407fcd1d455400b, entries=100, sequenceid=494, filesize=9.5 K 2023-04-27 01:55:12,187 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/0539201f82c64d8cb24f614f951336b9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0539201f82c64d8cb24f614f951336b9 2023-04-27 01:55:12,190 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0539201f82c64d8cb24f614f951336b9, entries=100, sequenceid=494, filesize=9.5 K 2023-04-27 01:55:12,191 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 1242ms, sequenceid=494, compaction requested=true 2023-04-27 01:55:12,191 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:12,191 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:12,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:12,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:12,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:12,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:12,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:12,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@604d51d5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:12,194 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:12,194 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:12,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:12,197 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:12,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:12,199 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:12,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:12,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:12,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:12,199 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:12,200 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:12,200 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:12,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:12,200 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:12,201 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:12,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:12,202 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,204 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:12,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:12,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:12,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:12,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:12,204 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:12,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:12,204 DEBUG [Thread-1535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:12,204 DEBUG [Thread-1535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:12,304 DEBUG [Thread-1535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:12,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:12,716 DEBUG [Thread-1527] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53921272 to 127.0.0.1:52731 2023-04-27 01:55:12,716 DEBUG [Thread-1527] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:12,733 DEBUG [Thread-1529] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13544701 to 127.0.0.1:52731 2023-04-27 01:55:12,733 DEBUG [Thread-1529] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:12,798 DEBUG [Thread-1525] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47772cff to 127.0.0.1:52731 2023-04-27 01:55:12,798 DEBUG [Thread-1525] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:12,798 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5003 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 4759 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 4761 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5003 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 4766 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 01:55:12,799 INFO [Listener at localhost/42495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 01:55:12,799 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23083cf2 to 127.0.0.1:52731 2023-04-27 01:55:12,799 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:12,800 INFO [Listener at localhost/42495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 01:55:12,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 01:55:12,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 01:55:12,803 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560512803"}]},"ts":"1682560512803"} 2023-04-27 01:55:12,804 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 01:55:12,805 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 01:55:12,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2bfc2234dd9d4cdcd80dce1fbcb604, UNASSIGN}] 2023-04-27 01:55:12,807 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2bfc2234dd9d4cdcd80dce1fbcb604, UNASSIGN 2023-04-27 01:55:12,808 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=6f2bfc2234dd9d4cdcd80dce1fbcb604, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:12,808 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560512808"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560512808"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560512808"}]},"ts":"1682560512808"} 2023-04-27 01:55:12,809 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:55:12,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 01:55:12,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6f2bfc2234dd9d4cdcd80dce1fbcb604, disabling compactions & flushes 2023-04-27 01:55:12,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. after waiting 0 ms 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:12,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6f2bfc2234dd9d4cdcd80dce1fbcb604 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=A 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=B 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6f2bfc2234dd9d4cdcd80dce1fbcb604, store=C 2023-04-27 01:55:12,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:12,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/88bd605f1ad047caa9d7616994edc263 2023-04-27 01:55:13,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 01:55:13,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/bece37a263fd43eaadd0b11da7673c46 2023-04-27 01:55:13,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 01:55:13,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/a24d06207f7b4ec4a36834c6b984c532 2023-04-27 01:55:13,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/A/88bd605f1ad047caa9d7616994edc263 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/88bd605f1ad047caa9d7616994edc263 2023-04-27 01:55:13,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/88bd605f1ad047caa9d7616994edc263, entries=150, sequenceid=502, filesize=11.9 K 2023-04-27 01:55:13,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/B/bece37a263fd43eaadd0b11da7673c46 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/bece37a263fd43eaadd0b11da7673c46 2023-04-27 01:55:13,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/bece37a263fd43eaadd0b11da7673c46, entries=150, sequenceid=502, filesize=11.9 K 2023-04-27 01:55:13,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/.tmp/C/a24d06207f7b4ec4a36834c6b984c532 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a24d06207f7b4ec4a36834c6b984c532 2023-04-27 01:55:13,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a24d06207f7b4ec4a36834c6b984c532, entries=150, sequenceid=502, filesize=11.9 K 2023-04-27 01:55:13,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 6f2bfc2234dd9d4cdcd80dce1fbcb604 in 838ms, sequenceid=502, compaction requested=true 2023-04-27 01:55:13,819 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/cc0cf281ea7e4795961f514e9cdb68cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/1235b0df27c7436dbb12099e4664751d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/a65f2d01dd314eddb56636a0ca5695ba, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c79f9245cd1e4af781277b1acd139384, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/ab7f853af104407a91a019605f13dcfe, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8174e527f8a046b1be220e064cdfc70f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/2d35d8b20f394d9b953ba18429bef6d8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8d9d09de27b24eb49a563a97cb6d5bb4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/94b959930bae4caba5694efca6eab200, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/fa9d04a68fce48cca8e853332837e4a4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9efaefbfccc84fca8daff24ac01545be, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/79c15d7441624ef8be4f07850d51fc96, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6718f374431e42d1a3fd132a77297db5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/02cd7bb49d884a30bb5b34652c5546ce, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/e88d51f083c7414592860a753493f620, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/514616ad436347388518d31aa3cfe733, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b387e549d87849cdb2cdf3d21ffbbcc8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b56bb71c6c704ae783435bf8d0ad05f4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/65a613d0cb824b758a5283552af99a1f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7f98d278178f4f19aee4fccb929325d6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b442182a4e3b4121aa533f1e53a30f91, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/de8c1d62cf6f4620a2194fd621c73053, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/0023b903a707406f97b4a9c4e82a5c6d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c93247e523b24625bf1dac2c48a64efa, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9fa8607404d741b2933ba38df5ce5317, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/d0b30c0ad19e49d4a3df50bd3f194fef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/919705293ec94433a5cee7bbed81e3a7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/eac35a97b9ab4472a3f9da07352db732, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6d3dd1f9edb646a29ad7799f0eeadafa] to archive 2023-04-27 01:55:13,820 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:55:13,821 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/cc0cf281ea7e4795961f514e9cdb68cc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/cc0cf281ea7e4795961f514e9cdb68cc 2023-04-27 01:55:13,822 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/1235b0df27c7436dbb12099e4664751d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/1235b0df27c7436dbb12099e4664751d 2023-04-27 01:55:13,823 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/a65f2d01dd314eddb56636a0ca5695ba to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/a65f2d01dd314eddb56636a0ca5695ba 2023-04-27 01:55:13,824 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c79f9245cd1e4af781277b1acd139384 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c79f9245cd1e4af781277b1acd139384 2023-04-27 01:55:13,825 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/ab7f853af104407a91a019605f13dcfe to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/ab7f853af104407a91a019605f13dcfe 2023-04-27 01:55:13,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8174e527f8a046b1be220e064cdfc70f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8174e527f8a046b1be220e064cdfc70f 2023-04-27 01:55:13,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/2d35d8b20f394d9b953ba18429bef6d8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/2d35d8b20f394d9b953ba18429bef6d8 2023-04-27 01:55:13,827 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8d9d09de27b24eb49a563a97cb6d5bb4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/8d9d09de27b24eb49a563a97cb6d5bb4 2023-04-27 01:55:13,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/94b959930bae4caba5694efca6eab200 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/94b959930bae4caba5694efca6eab200 2023-04-27 01:55:13,829 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/fa9d04a68fce48cca8e853332837e4a4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/fa9d04a68fce48cca8e853332837e4a4 2023-04-27 01:55:13,830 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9efaefbfccc84fca8daff24ac01545be to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9efaefbfccc84fca8daff24ac01545be 2023-04-27 01:55:13,831 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/79c15d7441624ef8be4f07850d51fc96 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/79c15d7441624ef8be4f07850d51fc96 2023-04-27 01:55:13,832 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6718f374431e42d1a3fd132a77297db5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6718f374431e42d1a3fd132a77297db5 2023-04-27 01:55:13,832 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/02cd7bb49d884a30bb5b34652c5546ce to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/02cd7bb49d884a30bb5b34652c5546ce 2023-04-27 01:55:13,833 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/e88d51f083c7414592860a753493f620 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/e88d51f083c7414592860a753493f620 2023-04-27 01:55:13,834 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/514616ad436347388518d31aa3cfe733 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/514616ad436347388518d31aa3cfe733 2023-04-27 01:55:13,835 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b387e549d87849cdb2cdf3d21ffbbcc8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b387e549d87849cdb2cdf3d21ffbbcc8 2023-04-27 01:55:13,835 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b56bb71c6c704ae783435bf8d0ad05f4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b56bb71c6c704ae783435bf8d0ad05f4 2023-04-27 01:55:13,836 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/65a613d0cb824b758a5283552af99a1f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/65a613d0cb824b758a5283552af99a1f 2023-04-27 01:55:13,837 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7f98d278178f4f19aee4fccb929325d6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7f98d278178f4f19aee4fccb929325d6 2023-04-27 01:55:13,838 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b442182a4e3b4121aa533f1e53a30f91 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/b442182a4e3b4121aa533f1e53a30f91 2023-04-27 01:55:13,839 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/de8c1d62cf6f4620a2194fd621c73053 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/de8c1d62cf6f4620a2194fd621c73053 2023-04-27 01:55:13,839 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/0023b903a707406f97b4a9c4e82a5c6d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/0023b903a707406f97b4a9c4e82a5c6d 2023-04-27 01:55:13,840 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c93247e523b24625bf1dac2c48a64efa to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/c93247e523b24625bf1dac2c48a64efa 2023-04-27 01:55:13,841 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9fa8607404d741b2933ba38df5ce5317 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/9fa8607404d741b2933ba38df5ce5317 2023-04-27 01:55:13,842 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/d0b30c0ad19e49d4a3df50bd3f194fef to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/d0b30c0ad19e49d4a3df50bd3f194fef 2023-04-27 01:55:13,843 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/919705293ec94433a5cee7bbed81e3a7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/919705293ec94433a5cee7bbed81e3a7 2023-04-27 01:55:13,844 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/eac35a97b9ab4472a3f9da07352db732 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/eac35a97b9ab4472a3f9da07352db732 2023-04-27 01:55:13,844 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6d3dd1f9edb646a29ad7799f0eeadafa to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/6d3dd1f9edb646a29ad7799f0eeadafa 2023-04-27 01:55:13,847 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f1a049bfee774c329128560498fd5ae1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/9fba5212580d4ed6a7be6dc8cd087cf5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b2f2716892a04bdd9874dbd2b4f915e3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2c199e40a8a84418b0f2fbc0a2e80e81, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/aa5c765d083e4004974da993863b6a8d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8da16e7bc6654cd992297869db0bb628, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b1cb7fef8f744991bd290bf235860a23, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d9fa4861df8c4595893518ceb318e265, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/90b961c41b4b46aa89b8a371c7f43a91, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2935a9470cc743589e45f6475a499a05, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/22698dc7806948409cb2138921d732b6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/df42961a90d5496dbae5e1f381ed0e64, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2a52887f4ab64fdf945deb704674c95b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/977fb7de596445bd96e2c3f9541df09f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d89f4477e1924de2a451e068ba48a4a7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f6e9ed9f7b5b4003b86efb52203a8fa7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/fd61c6bf3d524160adda21d1f4e28af5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/1ade6b1a85354a7d96d8f3185122b695, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/e50178aeffe241ecbe1a86dbd32ff159, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/7a168c95cff44556be73fab260a6feed, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/403ba414b4e6460baf8e51c95cd6c410, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/cd875da1c39241e99db3d312cb907f5a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/84618bd01d8543b1aede83656bb43023, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/907df09a6cf448c2a39e146b5f75b3e2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8428d978dc8d4dd294baaa4e41bb6665, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04d520ca08c443e393b23cbfaeb1e0d2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f44698c928984c0c882dbef277c6b021, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04ec62dd033e4eabbfeb408769ede386, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/387e533adadb4d4890d137be830b8878] to archive 2023-04-27 01:55:13,848 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:55:13,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f1a049bfee774c329128560498fd5ae1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f1a049bfee774c329128560498fd5ae1 2023-04-27 01:55:13,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/9fba5212580d4ed6a7be6dc8cd087cf5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/9fba5212580d4ed6a7be6dc8cd087cf5 2023-04-27 01:55:13,850 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b2f2716892a04bdd9874dbd2b4f915e3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b2f2716892a04bdd9874dbd2b4f915e3 2023-04-27 01:55:13,851 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2c199e40a8a84418b0f2fbc0a2e80e81 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2c199e40a8a84418b0f2fbc0a2e80e81 2023-04-27 01:55:13,852 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/aa5c765d083e4004974da993863b6a8d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/aa5c765d083e4004974da993863b6a8d 2023-04-27 01:55:13,853 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8da16e7bc6654cd992297869db0bb628 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8da16e7bc6654cd992297869db0bb628 2023-04-27 01:55:13,854 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b1cb7fef8f744991bd290bf235860a23 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/b1cb7fef8f744991bd290bf235860a23 2023-04-27 01:55:13,855 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d9fa4861df8c4595893518ceb318e265 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d9fa4861df8c4595893518ceb318e265 2023-04-27 01:55:13,855 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/90b961c41b4b46aa89b8a371c7f43a91 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/90b961c41b4b46aa89b8a371c7f43a91 2023-04-27 01:55:13,856 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2935a9470cc743589e45f6475a499a05 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2935a9470cc743589e45f6475a499a05 2023-04-27 01:55:13,857 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/22698dc7806948409cb2138921d732b6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/22698dc7806948409cb2138921d732b6 2023-04-27 01:55:13,858 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/df42961a90d5496dbae5e1f381ed0e64 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/df42961a90d5496dbae5e1f381ed0e64 2023-04-27 01:55:13,859 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2a52887f4ab64fdf945deb704674c95b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/2a52887f4ab64fdf945deb704674c95b 2023-04-27 01:55:13,860 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/977fb7de596445bd96e2c3f9541df09f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/977fb7de596445bd96e2c3f9541df09f 2023-04-27 01:55:13,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d89f4477e1924de2a451e068ba48a4a7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/d89f4477e1924de2a451e068ba48a4a7 2023-04-27 01:55:13,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f6e9ed9f7b5b4003b86efb52203a8fa7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f6e9ed9f7b5b4003b86efb52203a8fa7 2023-04-27 01:55:13,862 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/fd61c6bf3d524160adda21d1f4e28af5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/fd61c6bf3d524160adda21d1f4e28af5 2023-04-27 01:55:13,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/1ade6b1a85354a7d96d8f3185122b695 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/1ade6b1a85354a7d96d8f3185122b695 2023-04-27 01:55:13,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/e50178aeffe241ecbe1a86dbd32ff159 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/e50178aeffe241ecbe1a86dbd32ff159 2023-04-27 01:55:13,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/7a168c95cff44556be73fab260a6feed to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/7a168c95cff44556be73fab260a6feed 2023-04-27 01:55:13,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/403ba414b4e6460baf8e51c95cd6c410 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/403ba414b4e6460baf8e51c95cd6c410 2023-04-27 01:55:13,866 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/cd875da1c39241e99db3d312cb907f5a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/cd875da1c39241e99db3d312cb907f5a 2023-04-27 01:55:13,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/84618bd01d8543b1aede83656bb43023 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/84618bd01d8543b1aede83656bb43023 2023-04-27 01:55:13,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/907df09a6cf448c2a39e146b5f75b3e2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/907df09a6cf448c2a39e146b5f75b3e2 2023-04-27 01:55:13,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8428d978dc8d4dd294baaa4e41bb6665 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/8428d978dc8d4dd294baaa4e41bb6665 2023-04-27 01:55:13,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04d520ca08c443e393b23cbfaeb1e0d2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04d520ca08c443e393b23cbfaeb1e0d2 2023-04-27 01:55:13,870 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f44698c928984c0c882dbef277c6b021 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/f44698c928984c0c882dbef277c6b021 2023-04-27 01:55:13,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04ec62dd033e4eabbfeb408769ede386 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/04ec62dd033e4eabbfeb408769ede386 2023-04-27 01:55:13,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/387e533adadb4d4890d137be830b8878 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/387e533adadb4d4890d137be830b8878 2023-04-27 01:55:13,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/2273fdbb1a5448ac9090b25bdad1b7cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b83dcda6a3f447b1bd8fb10f38c4dc0b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c78680a039ae434b8ff9bc4ca1bf150f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/90b56aa9b31c4859b396b1aa4362d55c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f83ca41edc7a4b30831ab71da05a89ba, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b05ff8c78cfc461e92131a1aad51e757, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f4c8ff4d70344eca8798dbad8c65a8f1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/01b44cf2b52044f3ba93ec7e403d1263, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e6eb8a1373de4e8aaccfa8c73e1adcf7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/17e0b7d13b5942e0860a80daa362dd64, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e8737ce7f013445a91e0658930fecbee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b567be0f60e6424c9fe412e3f833e287, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0c5f6fdd4d584cc1b5fac387439e7b88, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/43589d3653be469fb8180a2349196cb1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/843497704bbd4a24bfb29dd9b33c7d7e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/84d1fc3034344fcd98a11d71b94a6c0d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/410bd96287db465fa9316c8b16cd6b68, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/64c390630342474b823216c7994d2374, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/58dfc421b90548269f142263e5f254a8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/ea401f4756a84b51b67f26afeb9872ba, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b53bfd2efffd46f0a3706279845775b7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/dceaf8a0a0b745a3a5087c53f52dbda3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/65ea97dcf2314fd5a1ae12e2d2e392f2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f0cb059d576d4cce9e8d2165d390e421, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a3b84539dfa047b68694134b2af88343, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/8677e24caa1e487cb6062d527c321371, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/44ca530c271f4fea8a98cb770d6250d6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c4217af7e2a54218b3a02824097800ea, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/9511d5155f3b4cc2bc8412a802cdc05b] to archive 2023-04-27 01:55:13,875 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:55:13,876 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/2273fdbb1a5448ac9090b25bdad1b7cc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/2273fdbb1a5448ac9090b25bdad1b7cc 2023-04-27 01:55:13,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b83dcda6a3f447b1bd8fb10f38c4dc0b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b83dcda6a3f447b1bd8fb10f38c4dc0b 2023-04-27 01:55:13,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c78680a039ae434b8ff9bc4ca1bf150f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c78680a039ae434b8ff9bc4ca1bf150f 2023-04-27 01:55:13,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/90b56aa9b31c4859b396b1aa4362d55c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/90b56aa9b31c4859b396b1aa4362d55c 2023-04-27 01:55:13,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f83ca41edc7a4b30831ab71da05a89ba to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f83ca41edc7a4b30831ab71da05a89ba 2023-04-27 01:55:13,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b05ff8c78cfc461e92131a1aad51e757 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b05ff8c78cfc461e92131a1aad51e757 2023-04-27 01:55:13,881 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f4c8ff4d70344eca8798dbad8c65a8f1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f4c8ff4d70344eca8798dbad8c65a8f1 2023-04-27 01:55:13,881 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/01b44cf2b52044f3ba93ec7e403d1263 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/01b44cf2b52044f3ba93ec7e403d1263 2023-04-27 01:55:13,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e6eb8a1373de4e8aaccfa8c73e1adcf7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e6eb8a1373de4e8aaccfa8c73e1adcf7 2023-04-27 01:55:13,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/17e0b7d13b5942e0860a80daa362dd64 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/17e0b7d13b5942e0860a80daa362dd64 2023-04-27 01:55:13,884 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e8737ce7f013445a91e0658930fecbee to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/e8737ce7f013445a91e0658930fecbee 2023-04-27 01:55:13,884 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b567be0f60e6424c9fe412e3f833e287 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b567be0f60e6424c9fe412e3f833e287 2023-04-27 01:55:13,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0c5f6fdd4d584cc1b5fac387439e7b88 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0c5f6fdd4d584cc1b5fac387439e7b88 2023-04-27 01:55:13,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/43589d3653be469fb8180a2349196cb1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/43589d3653be469fb8180a2349196cb1 2023-04-27 01:55:13,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/843497704bbd4a24bfb29dd9b33c7d7e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/843497704bbd4a24bfb29dd9b33c7d7e 2023-04-27 01:55:13,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/84d1fc3034344fcd98a11d71b94a6c0d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/84d1fc3034344fcd98a11d71b94a6c0d 2023-04-27 01:55:13,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/410bd96287db465fa9316c8b16cd6b68 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/410bd96287db465fa9316c8b16cd6b68 2023-04-27 01:55:13,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/64c390630342474b823216c7994d2374 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/64c390630342474b823216c7994d2374 2023-04-27 01:55:13,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/58dfc421b90548269f142263e5f254a8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/58dfc421b90548269f142263e5f254a8 2023-04-27 01:55:13,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/ea401f4756a84b51b67f26afeb9872ba to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/ea401f4756a84b51b67f26afeb9872ba 2023-04-27 01:55:13,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b53bfd2efffd46f0a3706279845775b7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/b53bfd2efffd46f0a3706279845775b7 2023-04-27 01:55:13,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/dceaf8a0a0b745a3a5087c53f52dbda3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/dceaf8a0a0b745a3a5087c53f52dbda3 2023-04-27 01:55:13,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/65ea97dcf2314fd5a1ae12e2d2e392f2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/65ea97dcf2314fd5a1ae12e2d2e392f2 2023-04-27 01:55:13,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f0cb059d576d4cce9e8d2165d390e421 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/f0cb059d576d4cce9e8d2165d390e421 2023-04-27 01:55:13,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a3b84539dfa047b68694134b2af88343 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a3b84539dfa047b68694134b2af88343 2023-04-27 01:55:13,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/8677e24caa1e487cb6062d527c321371 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/8677e24caa1e487cb6062d527c321371 2023-04-27 01:55:13,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/44ca530c271f4fea8a98cb770d6250d6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/44ca530c271f4fea8a98cb770d6250d6 2023-04-27 01:55:13,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c4217af7e2a54218b3a02824097800ea to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/c4217af7e2a54218b3a02824097800ea 2023-04-27 01:55:13,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/9511d5155f3b4cc2bc8412a802cdc05b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/9511d5155f3b4cc2bc8412a802cdc05b 2023-04-27 01:55:13,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/recovered.edits/505.seqid, newMaxSeqId=505, maxSeqId=1 2023-04-27 01:55:13,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604. 2023-04-27 01:55:13,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6f2bfc2234dd9d4cdcd80dce1fbcb604: 2023-04-27 01:55:13,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 01:55:13,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:13,907 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=6f2bfc2234dd9d4cdcd80dce1fbcb604, regionState=CLOSED 2023-04-27 01:55:13,907 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560513907"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560513907"}]},"ts":"1682560513907"} 2023-04-27 01:55:13,909 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-27 01:55:13,909 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 6f2bfc2234dd9d4cdcd80dce1fbcb604, server=jenkins-hbase4.apache.org,38085,1682560417985 in 1.0990 sec 2023-04-27 01:55:13,910 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-27 01:55:13,910 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6f2bfc2234dd9d4cdcd80dce1fbcb604, UNASSIGN in 1.1030 sec 2023-04-27 01:55:13,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560513911"}]},"ts":"1682560513911"} 2023-04-27 01:55:13,911 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 01:55:13,913 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 01:55:13,914 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1130 sec 2023-04-27 01:55:14,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 01:55:14,907 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-27 01:55:14,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 01:55:14,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:14,910 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:14,910 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:14,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-27 01:55:14,914 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:14,915 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/recovered.edits] 2023-04-27 01:55:14,918 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/4492735c3d0145ff8749e741dff09604 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/4492735c3d0145ff8749e741dff09604 2023-04-27 01:55:14,918 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7c106f4d6e2b4515b4f354ac524dcc9a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/7c106f4d6e2b4515b4f354ac524dcc9a 2023-04-27 01:55:14,919 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/88bd605f1ad047caa9d7616994edc263 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/88bd605f1ad047caa9d7616994edc263 2023-04-27 01:55:14,920 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/f7b26da067084d28b3284a4c105c7b45 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/A/f7b26da067084d28b3284a4c105c7b45 2023-04-27 01:55:14,922 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/5b64b744fd274475b08c7aa77daf7fd4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/5b64b744fd274475b08c7aa77daf7fd4 2023-04-27 01:55:14,923 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/67663244c16a48b2a8ea849697f7bda1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/67663244c16a48b2a8ea849697f7bda1 2023-04-27 01:55:14,924 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/712a6d2afd734aa8a407fcd1d455400b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/712a6d2afd734aa8a407fcd1d455400b 2023-04-27 01:55:14,925 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/bece37a263fd43eaadd0b11da7673c46 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/B/bece37a263fd43eaadd0b11da7673c46 2023-04-27 01:55:14,926 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0539201f82c64d8cb24f614f951336b9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/0539201f82c64d8cb24f614f951336b9 2023-04-27 01:55:14,927 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/1888176f8ac346f292dfcc48fc2bd7db to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/1888176f8ac346f292dfcc48fc2bd7db 2023-04-27 01:55:14,928 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/6173151927ff48139d413b46143046c1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/6173151927ff48139d413b46143046c1 2023-04-27 01:55:14,929 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a24d06207f7b4ec4a36834c6b984c532 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/C/a24d06207f7b4ec4a36834c6b984c532 2023-04-27 01:55:14,931 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/recovered.edits/505.seqid to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604/recovered.edits/505.seqid 2023-04-27 01:55:14,932 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/6f2bfc2234dd9d4cdcd80dce1fbcb604 2023-04-27 01:55:14,932 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:55:14,934 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:14,935 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 01:55:14,936 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 01:55:14,937 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:14,937 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 01:55:14,937 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682560514937"}]},"ts":"9223372036854775807"} 2023-04-27 01:55:14,939 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 01:55:14,939 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6f2bfc2234dd9d4cdcd80dce1fbcb604, NAME => 'TestAcidGuarantees,,1682560489977.6f2bfc2234dd9d4cdcd80dce1fbcb604.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 01:55:14,939 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 01:55:14,939 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682560514939"}]},"ts":"9223372036854775807"} 2023-04-27 01:55:14,940 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 01:55:14,942 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:14,942 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-04-27 01:55:15,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-27 01:55:15,011 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-27 01:55:15,023 INFO [Listener at localhost/42495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 224) - Thread LEAK? -, OpenFileDescriptor=557 (was 514) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=324 (was 329), ProcessCount=170 (was 170), AvailableMemoryMB=12006 (was 12038) 2023-04-27 01:55:15,035 INFO [Listener at localhost/42495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=557, MaxFileDescriptor=60000, SystemLoadAverage=324, ProcessCount=170, AvailableMemoryMB=12006 2023-04-27 01:55:15,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:55:15,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:55:15,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:15,039 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 01:55:15,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-27 01:55:15,039 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 01:55:15,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 01:55:15,040 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:15,041 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 empty. 2023-04-27 01:55:15,041 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:15,041 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:55:15,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 01:55:15,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 01:55:15,448 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:55:15,450 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b5f127509ba53373491a414d00be9fa1, NAME => 'TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp 2023-04-27 01:55:15,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 01:55:15,856 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:55:15,856 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b5f127509ba53373491a414d00be9fa1, disabling compactions & flushes 2023-04-27 01:55:15,856 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:15,856 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:15,856 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. after waiting 0 ms 2023-04-27 01:55:15,856 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:15,856 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:15,856 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:15,858 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 01:55:15,859 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560515859"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560515859"}]},"ts":"1682560515859"} 2023-04-27 01:55:15,860 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 01:55:15,861 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 01:55:15,861 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560515861"}]},"ts":"1682560515861"} 2023-04-27 01:55:15,862 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 01:55:15,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, ASSIGN}] 2023-04-27 01:55:15,867 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, ASSIGN 2023-04-27 01:55:15,868 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=false 2023-04-27 01:55:16,019 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:16,019 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560516019"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560516019"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560516019"}]},"ts":"1682560516019"} 2023-04-27 01:55:16,020 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:55:16,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 01:55:16,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:16,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b5f127509ba53373491a414d00be9fa1, NAME => 'TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:55:16,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:55:16,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,176 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,178 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A 2023-04-27 01:55:16,178 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A 2023-04-27 01:55:16,178 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:16,178 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b5f127509ba53373491a414d00be9fa1 columnFamilyName A 2023-04-27 01:55:16,178 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(310): Store=b5f127509ba53373491a414d00be9fa1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:16,179 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,179 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B 2023-04-27 01:55:16,180 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B 2023-04-27 01:55:16,180 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:16,180 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b5f127509ba53373491a414d00be9fa1 columnFamilyName B 2023-04-27 01:55:16,180 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(310): Store=b5f127509ba53373491a414d00be9fa1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:16,181 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,181 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C 2023-04-27 01:55:16,181 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C 2023-04-27 01:55:16,182 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:16,182 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b5f127509ba53373491a414d00be9fa1 columnFamilyName C 2023-04-27 01:55:16,182 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(310): Store=b5f127509ba53373491a414d00be9fa1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:16,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:16,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:55:16,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:16,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:55:16,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b5f127509ba53373491a414d00be9fa1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74867034, jitterRate=0.11560574173927307}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:55:16,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:16,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., pid=37, masterSystemTime=1682560516172 2023-04-27 01:55:16,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:16,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:16,189 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:16,189 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560516189"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560516189"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560516189"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560516189"}]},"ts":"1682560516189"} 2023-04-27 01:55:16,191 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-27 01:55:16,191 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 in 170 msec 2023-04-27 01:55:16,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-27 01:55:16,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, ASSIGN in 325 msec 2023-04-27 01:55:16,193 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 01:55:16,193 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560516193"}]},"ts":"1682560516193"} 2023-04-27 01:55:16,194 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 01:55:16,196 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 01:55:16,197 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-27 01:55:17,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 01:55:17,144 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-27 01:55:17,144 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x668bbba6 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:17,149 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038da2f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:17,150 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:17,152 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:17,153 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 01:55:17,154 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53198, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 01:55:17,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:55:17,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:55:17,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:17,565 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 01:55:17,566 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:55:17,566 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 01:55:17,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-27 01:55:17,570 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, REOPEN/MOVE}] 2023-04-27 01:55:17,571 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, REOPEN/MOVE 2023-04-27 01:55:17,571 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:17,571 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560517571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560517571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560517571"}]},"ts":"1682560517571"} 2023-04-27 01:55:17,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:55:17,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:17,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b5f127509ba53373491a414d00be9fa1, disabling compactions & flushes 2023-04-27 01:55:17,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:17,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:17,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. after waiting 0 ms 2023-04-27 01:55:17,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:17,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-27 01:55:17,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:17,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:17,728 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: b5f127509ba53373491a414d00be9fa1 to self. 2023-04-27 01:55:17,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:17,730 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=CLOSED 2023-04-27 01:55:17,730 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560517730"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560517730"}]},"ts":"1682560517730"} 2023-04-27 01:55:17,732 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-27 01:55:17,732 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 in 159 msec 2023-04-27 01:55:17,733 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=true 2023-04-27 01:55:17,883 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:17,883 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560517883"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560517883"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560517883"}]},"ts":"1682560517883"} 2023-04-27 01:55:17,884 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:55:18,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b5f127509ba53373491a414d00be9fa1, NAME => 'TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:55:18,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:55:18,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,040 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,041 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A 2023-04-27 01:55:18,041 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A 2023-04-27 01:55:18,042 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:18,042 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b5f127509ba53373491a414d00be9fa1 columnFamilyName A 2023-04-27 01:55:18,043 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(310): Store=b5f127509ba53373491a414d00be9fa1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:18,043 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,044 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B 2023-04-27 01:55:18,044 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B 2023-04-27 01:55:18,044 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:18,044 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b5f127509ba53373491a414d00be9fa1 columnFamilyName B 2023-04-27 01:55:18,045 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(310): Store=b5f127509ba53373491a414d00be9fa1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:18,045 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,045 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C 2023-04-27 01:55:18,045 DEBUG [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C 2023-04-27 01:55:18,045 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:18,045 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b5f127509ba53373491a414d00be9fa1 columnFamilyName C 2023-04-27 01:55:18,046 INFO [StoreOpener-b5f127509ba53373491a414d00be9fa1-1] regionserver.HStore(310): Store=b5f127509ba53373491a414d00be9fa1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:18,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:55:18,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b5f127509ba53373491a414d00be9fa1; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68279536, jitterRate=0.017444372177124023}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:55:18,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:18,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., pid=42, masterSystemTime=1682560518036 2023-04-27 01:55:18,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,053 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,053 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560518052"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560518052"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560518052"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560518052"}]},"ts":"1682560518052"} 2023-04-27 01:55:18,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-27 01:55:18,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 in 170 msec 2023-04-27 01:55:18,055 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-27 01:55:18,055 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, REOPEN/MOVE in 484 msec 2023-04-27 01:55:18,057 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-27 01:55:18,057 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-27 01:55:18,059 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-27 01:55:18,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-27 01:55:18,060 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x7a9450ee to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,065 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41798291, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,066 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x494529f9 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,069 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@118e131f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,069 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x4d800352 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,072 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2deaf1fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,072 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x2329054b to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,076 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e7e18ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,076 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x30951255 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,081 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b7d4a51, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,081 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x4335c1f2 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,084 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@722b2407, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,084 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x6bbc9fc8 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,087 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66bf91c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,087 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x678b4c3d to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,093 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6846dddb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,093 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x43736f95 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,097 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ea43962, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,098 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x72eaaf6e to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:18,101 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2abd7961, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:18,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:18,112 DEBUG [hconnection-0x75035bad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,113 DEBUG [hconnection-0x77fc8f6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,113 DEBUG [hconnection-0x560cbf86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,113 DEBUG [hconnection-0x78625b03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,113 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,114 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,114 DEBUG [hconnection-0x5b2aa711-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,114 DEBUG [hconnection-0x5a398d8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:18,116 DEBUG [hconnection-0xc3bf1b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,116 DEBUG [hconnection-0x3f99f635-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,121 DEBUG [hconnection-0x646a9268-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,121 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40264, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,121 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40294, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,121 DEBUG [hconnection-0x48163b91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:18,121 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,123 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,123 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:18,124 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:18,124 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,123 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:18,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:18,126 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:18,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,133 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:18,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:18,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:18,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:18,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:18,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:18,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:18,140 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:18,141 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,141 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:18,141 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:18,141 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:18,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:18,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:18,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:18,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:18,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:18,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:18,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:18,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:18,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:18,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560578154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560518153, max timestamp=1682560518154Num uniques 859; 2023-04-27 01:55:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:55:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560518153, max timestamp=1682560518154Num uniques 859; 2023-04-27 01:55:18,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:55:18,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560518153, max timestamp=1682560518154Num uniques 859; 2023-04-27 01:55:18,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:55:18,162 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,162 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,163 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eb096b3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:18,165 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:18,165 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,166 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:18,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:18,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:18,169 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:18,169 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,169 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,169 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:18,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:18,172 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:18,172 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:18,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:18,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:18,173 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:18,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,180 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560578178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560578178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560578180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560578181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:18,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:18,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,186 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:18,186 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:18,186 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:18,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,190 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272ca915e71b314cb88753b286d6b99dec to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272ca915e71b314cb88753b286d6b99dec 2023-04-27 01:55:18,200 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/ec5e3506e6e24c418949284007b581af 2023-04-27 01:55:18,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560578259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560578282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560578282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560578282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560578282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,286 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:18,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:18,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:18,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:18,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:18,292 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:18,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:18,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:18,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,293 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,294 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,294 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:18,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:18,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:18,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:18,299 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:18,300 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371532ab[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:18,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:18,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,306 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:18,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:18,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:18,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:18,309 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:18,312 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:18,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:18,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:18,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:18,313 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:18,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:18,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:18,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,318 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:18,318 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:18,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,418 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:18,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:18,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:18,421 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:18,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:18,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:18,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:18,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:18,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:18,429 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6962277a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:18,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:18,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,435 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:18,435 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:18,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:18,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:18,439 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:18,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:18,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:18,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:18,439 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:18,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,442 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:18,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:18,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,446 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,446 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:18,446 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560578466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560578486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560578486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560578486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560578486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,546 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:18,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:18,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:18,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:18,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:18,553 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:18,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:18,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:18,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:18,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:18,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:18,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@632b107a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:18,576 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:18,576 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:18,582 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,582 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:18,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:18,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:18,591 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:18,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:18,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:18,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:18,591 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:18,592 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,592 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:18,592 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:18,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,597 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,597 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:18,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:18,598 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:18,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:18,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:18,598 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:18,598 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:18,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:18,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:18,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/c7e897701be5431493eca8bc8b1f722c 2023-04-27 01:55:18,635 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/0c8223194e6a46578701919837eb66db 2023-04-27 01:55:18,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/ec5e3506e6e24c418949284007b581af as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ec5e3506e6e24c418949284007b581af 2023-04-27 01:55:18,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ec5e3506e6e24c418949284007b581af, entries=200, sequenceid=17, filesize=31.9 K 2023-04-27 01:55:18,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/c7e897701be5431493eca8bc8b1f722c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c7e897701be5431493eca8bc8b1f722c 2023-04-27 01:55:18,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c7e897701be5431493eca8bc8b1f722c, entries=150, sequenceid=17, filesize=11.6 K 2023-04-27 01:55:18,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/0c8223194e6a46578701919837eb66db as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0c8223194e6a46578701919837eb66db 2023-04-27 01:55:18,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0c8223194e6a46578701919837eb66db, entries=150, sequenceid=17, filesize=11.6 K 2023-04-27 01:55:18,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=120.76 KB/123660 for b5f127509ba53373491a414d00be9fa1 in 512ms, sequenceid=17, compaction requested=false 2023-04-27 01:55:18,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:18,698 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:18,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:18,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:18,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:18,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:18,702 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:18,702 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,702 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:18,702 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:18,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,703 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:18,705 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:18,705 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,705 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:18,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:18,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:18,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:18,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:18,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:18,707 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-04-27 01:55:18,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:18,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:18,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:18,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:18,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:18,708 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:18,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:18,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560578795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560578796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560578797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560578797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560578798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560578902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560578903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560578905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560578905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:18,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:18,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560578907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560579110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560579111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560579113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560579113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560579114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,124 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042784e880608fa54542b53e4e6f43323397 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042784e880608fa54542b53e4e6f43323397 2023-04-27 01:55:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560579416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560579416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560579416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560579416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:19,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560579417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,433 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 01:55:19,529 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/7d2fe5470ce44e398544c811faa51af3 2023-04-27 01:55:19,547 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/dbaef951a7b24074ab4020b9a78fcc9f 2023-04-27 01:55:19,557 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/9272833f438545ccaeeb334bd11a77ee 2023-04-27 01:55:19,561 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/7d2fe5470ce44e398544c811faa51af3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7d2fe5470ce44e398544c811faa51af3 2023-04-27 01:55:19,569 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7d2fe5470ce44e398544c811faa51af3, entries=150, sequenceid=52, filesize=25.1 K 2023-04-27 01:55:19,571 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/dbaef951a7b24074ab4020b9a78fcc9f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/dbaef951a7b24074ab4020b9a78fcc9f 2023-04-27 01:55:19,576 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/dbaef951a7b24074ab4020b9a78fcc9f, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 01:55:19,576 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/9272833f438545ccaeeb334bd11a77ee as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9272833f438545ccaeeb334bd11a77ee 2023-04-27 01:55:19,580 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9272833f438545ccaeeb334bd11a77ee, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 01:55:19,580 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for b5f127509ba53373491a414d00be9fa1 in 873ms, sequenceid=52, compaction requested=false 2023-04-27 01:55:19,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:19,580 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:19,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a407a2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:19,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:19,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:19,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:19,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:19,586 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,586 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:19,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,587 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:19,587 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:19,588 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:19,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:19,590 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:19,590 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:19,590 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:19,590 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:19,590 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:19,593 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:19,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,600 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,600 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:19,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:19,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:19,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:19,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:19,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:19,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:19,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:19,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:19,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:19,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,605 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:19,605 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:19,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:19,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:19,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:19,705 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:19,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:19,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:19,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:19,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:19,723 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:19,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:19,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:19,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:19,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:19,724 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,724 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:19,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:19,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:19,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:19,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:19,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:19,737 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:19,737 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:19,737 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:19,737 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:19,738 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:19,738 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:19,738 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:19,738 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:19,738 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:19,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:20,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560579996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560579997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560579996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560580008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560580009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560580110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560580110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560580112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560580123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560580124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,169 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427b68f7117924342c7b19620567505f273 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b68f7117924342c7b19620567505f273 2023-04-27 01:55:20,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560580317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560580318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560580319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560580336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560580336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,587 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=64, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/20decab11cfb4dc2a5f98ba664eee04e 2023-04-27 01:55:20,609 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/96e0f3aaabca4d18bde74c5a287dcb27 2023-04-27 01:55:20,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560580625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560580629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560580629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,644 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/deeadc77274844aaaf4ac693b9e2acbd 2023-04-27 01:55:20,649 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/20decab11cfb4dc2a5f98ba664eee04e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/20decab11cfb4dc2a5f98ba664eee04e 2023-04-27 01:55:20,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560580643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560580643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,654 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/20decab11cfb4dc2a5f98ba664eee04e, entries=100, sequenceid=64, filesize=18.4 K 2023-04-27 01:55:20,655 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/96e0f3aaabca4d18bde74c5a287dcb27 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/96e0f3aaabca4d18bde74c5a287dcb27 2023-04-27 01:55:20,661 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/96e0f3aaabca4d18bde74c5a287dcb27, entries=100, sequenceid=64, filesize=9.3 K 2023-04-27 01:55:20,662 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/deeadc77274844aaaf4ac693b9e2acbd as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/deeadc77274844aaaf4ac693b9e2acbd 2023-04-27 01:55:20,666 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/deeadc77274844aaaf4ac693b9e2acbd, entries=100, sequenceid=64, filesize=9.3 K 2023-04-27 01:55:20,667 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b5f127509ba53373491a414d00be9fa1 in 930ms, sequenceid=64, compaction requested=true 2023-04-27 01:55:20,667 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:20,667 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:20,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:20,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:20,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:20,670 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,670 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,670 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,670 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:20,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773ae057[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:20,672 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:20,672 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,673 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:20,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:20,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:20,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:20,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:20,676 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:20,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:20,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:20,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:20,676 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:20,677 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,677 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:20,677 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:20,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:20,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:20,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:20,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:20,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:20,683 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:20,683 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:20,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:20,784 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:20,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:20,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:20,788 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:20,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:20,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:20,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:20,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:20,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:20,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:20,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:20,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:20,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:20,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:20,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:20,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:20,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:20,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:20,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:20,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:20,796 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:20,796 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:20,797 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:20,797 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:20,797 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:20,797 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:21,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:21,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560581151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560581152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560581153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560581154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560581158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,251 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042756aff53db9174299b9fdffc06cd7d3d0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042756aff53db9174299b9fdffc06cd7d3d0 2023-04-27 01:55:21,257 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/cc948ad734164dd3aec7b33c67a0bf2d 2023-04-27 01:55:21,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560581255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,266 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/110d6b5209a246b2afdd0bbcb754757a 2023-04-27 01:55:21,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560581261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560581262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560581463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560581467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560581468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,677 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/0d0a2190e3b34ee6aa302f6907bbb2d7 2023-04-27 01:55:21,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/cc948ad734164dd3aec7b33c67a0bf2d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/cc948ad734164dd3aec7b33c67a0bf2d 2023-04-27 01:55:21,684 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/cc948ad734164dd3aec7b33c67a0bf2d, entries=150, sequenceid=84, filesize=25.1 K 2023-04-27 01:55:21,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/110d6b5209a246b2afdd0bbcb754757a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/110d6b5209a246b2afdd0bbcb754757a 2023-04-27 01:55:21,689 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/110d6b5209a246b2afdd0bbcb754757a, entries=150, sequenceid=84, filesize=11.6 K 2023-04-27 01:55:21,689 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/0d0a2190e3b34ee6aa302f6907bbb2d7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0d0a2190e3b34ee6aa302f6907bbb2d7 2023-04-27 01:55:21,692 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0d0a2190e3b34ee6aa302f6907bbb2d7, entries=150, sequenceid=84, filesize=11.6 K 2023-04-27 01:55:21,693 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b5f127509ba53373491a414d00be9fa1 in 897ms, sequenceid=84, compaction requested=true 2023-04-27 01:55:21,693 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:21,693 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:21,694 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:21,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f1b403d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:21,696 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:21,696 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,697 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:21,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:21,707 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:21,709 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:21,709 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:21,709 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:21,709 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:21,709 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:21,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:21,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,712 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,714 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:21,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:21,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:21,714 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:21,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:21,714 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:21,714 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:21,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:21,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:21,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:55:21,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:21,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:21,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:21,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:21,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:21,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:21,814 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:21,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:21,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:21,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:21,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:21,818 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:21,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:21,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:21,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:21,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:21,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:21,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:21,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:21,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:21,822 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:21,822 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:21,822 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:21,822 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:21,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560581815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,825 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,825 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:21,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34082964[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:21,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:21,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:21,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:21,829 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560581824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560581824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:21,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:21,834 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:21,835 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:21,835 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:21,835 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:21,835 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:21,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,836 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,837 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:21,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:21,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,838 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:21,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:21,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:21,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:21,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:21,843 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:21,843 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:21,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560581925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560581933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:21,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560581933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,943 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:21,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:21,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:21,947 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:21,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:21,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:21,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:21,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:21,949 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:21,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:21,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:21,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:21,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:21,952 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:21,952 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:21,952 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:21,952 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:21,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,953 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,954 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,954 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:21,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dd8b150[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:21,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:21,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,956 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:21,956 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:21,956 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:21,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,957 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:21,962 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:21,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:21,966 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:21,966 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:21,966 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:21,966 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:21,966 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:21,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:21,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:21,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,971 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:21,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:21,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:21,975 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,975 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:21,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:21,975 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:21,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:21,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,075 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:22,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:22,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:22,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:22,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:22,086 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:22,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:22,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:22,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,090 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,090 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:22,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:22,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:22,091 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:22,091 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,091 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:22,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:22,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:22,092 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:22,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,098 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:22,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d40cee4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:22,100 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:22,100 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,101 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:22,102 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:22,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:22,103 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:22,103 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,103 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,103 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:22,106 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:22,106 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:22,106 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:22,106 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:22,106 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:22,107 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,107 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,107 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,108 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,115 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,115 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,115 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:22,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:22,115 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:22,115 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,115 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,115 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:22,115 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,116 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:22,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,116 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,116 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,116 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560582130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560582137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560582137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560582162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560582163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,195 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042776a4ef6e46ec4004a3cf7f2890c15156 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042776a4ef6e46ec4004a3cf7f2890c15156 2023-04-27 01:55:22,206 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/a30fb2f08c2c419f85efcfb6f91fa0cd 2023-04-27 01:55:22,216 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:22,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:22,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:22,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:22,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:22,225 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:22,225 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,225 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:22,225 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:22,226 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,226 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,227 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,227 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,227 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,227 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:22,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:22,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:22,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:22,232 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,232 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:22,232 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:22,232 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:22,234 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,234 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,234 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,234 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:22,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88f77d9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:22,236 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:22,236 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,237 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:22,237 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,237 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:22,239 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:22,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,239 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:22,242 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:22,242 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:22,242 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:22,242 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:22,242 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:22,243 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,243 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,243 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,245 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,248 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,248 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:22,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:22,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:22,249 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:22,249 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:22,349 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:22,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:22,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:22,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:22,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:22,352 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:22,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:22,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:22,353 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,353 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,354 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,354 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,354 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,354 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:22,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:22,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:22,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:22,355 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:22,355 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,355 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:22,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:22,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:22,356 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:22,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d392ca5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:22,360 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:22,360 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:22,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:22,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:22,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:22,365 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:22,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:22,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:22,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:22,365 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:22,366 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,366 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,367 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,367 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,368 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,371 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:22,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,371 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,371 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:22,372 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:22,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:22,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560582440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560582443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560582443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,472 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:22,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:22,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:22,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:22,475 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:22,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:22,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:22,476 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,476 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,478 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,478 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,478 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,479 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:22,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:22,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:22,480 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,480 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:22,480 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:22,480 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:22,480 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:22,482 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,482 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,483 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,483 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:22,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45534063[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:22,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:22,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,485 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:22,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:22,486 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,486 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:22,486 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:22,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:22,489 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:22,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:22,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:22,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:22,490 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:22,491 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,491 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,491 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,493 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,498 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,498 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,498 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:22,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:22,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:22,498 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:22,498 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:22,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,501 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,501 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,501 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,501 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,501 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,501 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,501 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,598 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:22,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:22,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:22,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:22,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:22,602 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:22,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:22,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:22,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,602 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:22,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:22,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:22,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:22,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:22,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:22,605 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,605 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:22,605 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:22,605 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:22,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:22,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:22,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:22,607 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:22,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3f91c3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:22,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:22,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:22,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:22,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:22,611 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:22,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:22,615 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:22,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:22,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:22,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:22,615 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:22,617 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,617 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,617 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:22,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:22,618 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,619 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:22,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:22,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:22,621 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:22,621 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:22,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:22,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,621 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:22,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/c59ffb094e944994b9176a28391edee1 2023-04-27 01:55:22,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/b1a399f93ec947ad9d92fae5dd80c371 2023-04-27 01:55:22,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/a30fb2f08c2c419f85efcfb6f91fa0cd as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a30fb2f08c2c419f85efcfb6f91fa0cd 2023-04-27 01:55:22,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a30fb2f08c2c419f85efcfb6f91fa0cd, entries=200, sequenceid=97, filesize=31.9 K 2023-04-27 01:55:22,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/c59ffb094e944994b9176a28391edee1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c59ffb094e944994b9176a28391edee1 2023-04-27 01:55:22,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c59ffb094e944994b9176a28391edee1, entries=150, sequenceid=97, filesize=11.6 K 2023-04-27 01:55:22,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/b1a399f93ec947ad9d92fae5dd80c371 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/b1a399f93ec947ad9d92fae5dd80c371 2023-04-27 01:55:22,665 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/b1a399f93ec947ad9d92fae5dd80c371, entries=150, sequenceid=97, filesize=11.6 K 2023-04-27 01:55:22,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for b5f127509ba53373491a414d00be9fa1 in 893ms, sequenceid=97, compaction requested=true 2023-04-27 01:55:22,666 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:22,666 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:22,666 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:55:22,666 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:55:22,666 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:22,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:22,668 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 135662 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:55:22,668 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/A is initiating minor compaction (all files) 2023-04-27 01:55:22,668 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:55:22,668 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/A in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,668 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/B is initiating minor compaction (all files) 2023-04-27 01:55:22,668 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/B in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,668 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ec5e3506e6e24c418949284007b581af, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7d2fe5470ce44e398544c811faa51af3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/20decab11cfb4dc2a5f98ba664eee04e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/cc948ad734164dd3aec7b33c67a0bf2d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a30fb2f08c2c419f85efcfb6f91fa0cd] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=132.5 K 2023-04-27 01:55:22,669 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c7e897701be5431493eca8bc8b1f722c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/dbaef951a7b24074ab4020b9a78fcc9f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/96e0f3aaabca4d18bde74c5a287dcb27, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/110d6b5209a246b2afdd0bbcb754757a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c59ffb094e944994b9176a28391edee1] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=55.7 K 2023-04-27 01:55:22,669 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting ec5e3506e6e24c418949284007b581af, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682560518136 2023-04-27 01:55:22,669 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c7e897701be5431493eca8bc8b1f722c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682560518136 2023-04-27 01:55:22,671 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 7d2fe5470ce44e398544c811faa51af3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682560518177 2023-04-27 01:55:22,671 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting dbaef951a7b24074ab4020b9a78fcc9f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682560518177 2023-04-27 01:55:22,672 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 20decab11cfb4dc2a5f98ba664eee04e, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682560518796 2023-04-27 01:55:22,672 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 96e0f3aaabca4d18bde74c5a287dcb27, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682560518796 2023-04-27 01:55:22,672 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cc948ad734164dd3aec7b33c67a0bf2d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682560519992 2023-04-27 01:55:22,673 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 110d6b5209a246b2afdd0bbcb754757a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682560519992 2023-04-27 01:55:22,673 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a30fb2f08c2c419f85efcfb6f91fa0cd, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682560521146 2023-04-27 01:55:22,673 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c59ffb094e944994b9176a28391edee1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682560521146 2023-04-27 01:55:22,697 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#B#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:22,714 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#A#compaction#306 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:22,722 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:22,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:22,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:22,726 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/afb242fc39f842ec87b51d1194fbb246 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/afb242fc39f842ec87b51d1194fbb246 2023-04-27 01:55:22,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:22,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:22,726 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:22,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:22,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:22,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,728 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,728 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:22,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:22,728 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,728 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:22,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:22,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:22,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:22,730 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b5f127509ba53373491a414d00be9fa1/B of b5f127509ba53373491a414d00be9fa1 into afb242fc39f842ec87b51d1194fbb246(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:22,730 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:22,730 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/B, priority=11, startTime=1682560522666; duration=0sec 2023-04-27 01:55:22,730 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:22,731 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:55:22,732 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:55:22,732 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/C is initiating minor compaction (all files) 2023-04-27 01:55:22,732 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/C in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,732 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:22,732 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:22,732 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:22,732 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:22,732 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0c8223194e6a46578701919837eb66db, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9272833f438545ccaeeb334bd11a77ee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/deeadc77274844aaaf4ac693b9e2acbd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0d0a2190e3b34ee6aa302f6907bbb2d7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/b1a399f93ec947ad9d92fae5dd80c371] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=55.7 K 2023-04-27 01:55:22,732 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:22,732 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:22,732 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 0c8223194e6a46578701919837eb66db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682560518136 2023-04-27 01:55:22,733 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:22,733 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:22,733 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:22,733 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:22,733 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:22,733 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9272833f438545ccaeeb334bd11a77ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682560518177 2023-04-27 01:55:22,733 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting deeadc77274844aaaf4ac693b9e2acbd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682560518796 2023-04-27 01:55:22,733 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 0d0a2190e3b34ee6aa302f6907bbb2d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682560519992 2023-04-27 01:55:22,734 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b1a399f93ec947ad9d92fae5dd80c371, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682560521146 2023-04-27 01:55:22,741 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#C#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:22,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:22,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:22,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560582961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560582964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:22,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:22,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560582965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560583069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560583069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560583072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,154 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042781c95994a3d6459bbfd2ae2ab6435172 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042781c95994a3d6459bbfd2ae2ab6435172 2023-04-27 01:55:23,155 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/698aa6533a7b4aa9b0fb0d56dfa8e16a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/698aa6533a7b4aa9b0fb0d56dfa8e16a 2023-04-27 01:55:23,160 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b5f127509ba53373491a414d00be9fa1/C of b5f127509ba53373491a414d00be9fa1 into 698aa6533a7b4aa9b0fb0d56dfa8e16a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:23,161 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:23,161 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/C, priority=11, startTime=1682560522667; duration=0sec 2023-04-27 01:55:23,161 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:23,162 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/d502d235b57342a6b51d5bcf54961b35 2023-04-27 01:55:23,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560583276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560583276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560583276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,574 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/48219db5b6a248e6875abab624b67838 2023-04-27 01:55:23,584 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/09bd3b4b91d04befaf89d9c7a897b920 2023-04-27 01:55:23,588 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/d502d235b57342a6b51d5bcf54961b35 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d502d235b57342a6b51d5bcf54961b35 2023-04-27 01:55:23,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560583582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560583583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560583583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,592 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d502d235b57342a6b51d5bcf54961b35, entries=150, sequenceid=118, filesize=25.1 K 2023-04-27 01:55:23,593 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/48219db5b6a248e6875abab624b67838 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/48219db5b6a248e6875abab624b67838 2023-04-27 01:55:23,596 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/48219db5b6a248e6875abab624b67838, entries=150, sequenceid=118, filesize=11.6 K 2023-04-27 01:55:23,596 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/09bd3b4b91d04befaf89d9c7a897b920 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/09bd3b4b91d04befaf89d9c7a897b920 2023-04-27 01:55:23,599 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/09bd3b4b91d04befaf89d9c7a897b920, entries=150, sequenceid=118, filesize=11.6 K 2023-04-27 01:55:23,600 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b5f127509ba53373491a414d00be9fa1 in 868ms, sequenceid=118, compaction requested=false 2023-04-27 01:55:23,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:23,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:23,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,602 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dfd1c62[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:23,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:23,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:23,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:23,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:23,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:23,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:23,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:23,610 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:23,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:23,612 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:23,612 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:23,612 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:23,612 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:23,612 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:23,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:23,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:23,614 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,617 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:23,617 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:23,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:23,617 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:23,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:23,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:23,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:23,617 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:23,617 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:23,617 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,617 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:23,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:23,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:23,618 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,717 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:23,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:23,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:23,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:23,721 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:23,721 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:23,721 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:23,721 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:23,722 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,722 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:23,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:23,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:23,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:23,723 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:23,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:23,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:23,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:23,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:23,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:23,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:23,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:23,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:23,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:23,725 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:23,725 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:23,942 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/b8946eafdfdf48acb509eae9828d950a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b8946eafdfdf48acb509eae9828d950a 2023-04-27 01:55:23,946 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b5f127509ba53373491a414d00be9fa1/A of b5f127509ba53373491a414d00be9fa1 into b8946eafdfdf48acb509eae9828d950a(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 01:55:23,946 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:23,946 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/A, priority=11, startTime=1682560522666; duration=1sec 2023-04-27 01:55:23,946 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:24,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:24,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:24,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560584117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560584118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560584119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,142 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042792ca08829fb440ee97a35c95142f2f2d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042792ca08829fb440ee97a35c95142f2f2d 2023-04-27 01:55:24,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560584180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,183 DEBUG [Thread-2176] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:24,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560584188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,193 DEBUG [Thread-2170] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:24,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560584222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560584224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560584226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560584424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560584430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560584430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,547 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/8cd75142a24b4ee4bce2e1119bdd9027 2023-04-27 01:55:24,556 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/9ab0be9be2784140b681bf03d4ed25f9 2023-04-27 01:55:24,573 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/d881e1c0f1d74093a79249f726db49be 2023-04-27 01:55:24,578 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/8cd75142a24b4ee4bce2e1119bdd9027 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/8cd75142a24b4ee4bce2e1119bdd9027 2023-04-27 01:55:24,585 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/8cd75142a24b4ee4bce2e1119bdd9027, entries=150, sequenceid=131, filesize=25.2 K 2023-04-27 01:55:24,585 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/9ab0be9be2784140b681bf03d4ed25f9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ab0be9be2784140b681bf03d4ed25f9 2023-04-27 01:55:24,589 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ab0be9be2784140b681bf03d4ed25f9, entries=150, sequenceid=131, filesize=11.7 K 2023-04-27 01:55:24,589 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/d881e1c0f1d74093a79249f726db49be as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d881e1c0f1d74093a79249f726db49be 2023-04-27 01:55:24,593 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d881e1c0f1d74093a79249f726db49be, entries=150, sequenceid=131, filesize=11.7 K 2023-04-27 01:55:24,594 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b5f127509ba53373491a414d00be9fa1 in 869ms, sequenceid=131, compaction requested=true 2023-04-27 01:55:24,594 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:24,594 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:24,597 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:24,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:24,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5915597c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:24,599 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:24,599 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,600 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:24,600 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,600 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:24,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:24,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:24,601 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:24,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:24,604 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:24,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:24,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:24,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:24,604 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:24,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:24,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:24,606 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:24,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:24,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:24,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:24,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:24,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:24,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:24,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:24,610 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:24,610 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:24,610 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:24,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:24,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:24,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,612 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:24,612 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:24,710 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:24,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:24,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:24,714 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:24,714 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:24,714 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:24,714 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:24,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:24,719 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:24,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:24,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:24,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:24,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:24,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:24,721 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:24,721 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:24,721 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:24,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:24,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:24,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:24,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:24,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:24,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:24,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:24,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:24,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560584747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560584750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560584753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560584854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560584854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:24,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:24,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560584862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560585060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560585062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560585067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,139 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304279cbeacc833f3476a86067980c9d1d092 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279cbeacc833f3476a86067980c9d1d092 2023-04-27 01:55:25,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560585367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560585368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560585374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,550 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/7a0843ed25594282bfbe4b83c1921d19 2023-04-27 01:55:25,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560585876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560585878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:25,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560585881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:25,962 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/9ef76e5d90b741eaacf273a47b9b5914 2023-04-27 01:55:25,973 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/25de986d2ea545b5b9a9ff6c9e21a3ca 2023-04-27 01:55:25,985 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/7a0843ed25594282bfbe4b83c1921d19 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7a0843ed25594282bfbe4b83c1921d19 2023-04-27 01:55:25,990 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7a0843ed25594282bfbe4b83c1921d19, entries=150, sequenceid=152, filesize=25.3 K 2023-04-27 01:55:25,991 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/9ef76e5d90b741eaacf273a47b9b5914 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ef76e5d90b741eaacf273a47b9b5914 2023-04-27 01:55:25,995 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ef76e5d90b741eaacf273a47b9b5914, entries=150, sequenceid=152, filesize=11.8 K 2023-04-27 01:55:25,996 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/25de986d2ea545b5b9a9ff6c9e21a3ca as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/25de986d2ea545b5b9a9ff6c9e21a3ca 2023-04-27 01:55:25,999 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/25de986d2ea545b5b9a9ff6c9e21a3ca, entries=150, sequenceid=152, filesize=11.8 K 2023-04-27 01:55:26,000 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b5f127509ba53373491a414d00be9fa1 in 1279ms, sequenceid=152, compaction requested=true 2023-04-27 01:55:26,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:26,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:26,001 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:26,002 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:26,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a5a65[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:26,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:26,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:26,005 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,005 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:26,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:26,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:26,009 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:26,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:26,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:26,010 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:26,010 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:26,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,013 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,014 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:26,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:26,016 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:26,017 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:26,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,017 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,117 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:26,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:26,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:26,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:26,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:26,121 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:26,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:26,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:26,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:26,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,123 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:26,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,124 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:26,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:26,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:26,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:26,133 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:26,133 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:26,133 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:26,133 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:26,133 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:26,133 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:26,133 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:26,134 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:26,134 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:26,575 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042798f5ae42687f46938ab2850c693da46d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042798f5ae42687f46938ab2850c693da46d 2023-04-27 01:55:26,602 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=164, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/c2592bbbc225433c9ab663291a8e83f3 2023-04-27 01:55:26,614 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/85e4656b645441dbbe74ea29904f5eb5 2023-04-27 01:55:26,630 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/04fa7ee4836c4de288854fa655276e77 2023-04-27 01:55:26,634 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/c2592bbbc225433c9ab663291a8e83f3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/c2592bbbc225433c9ab663291a8e83f3 2023-04-27 01:55:26,638 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/c2592bbbc225433c9ab663291a8e83f3, entries=100, sequenceid=164, filesize=18.5 K 2023-04-27 01:55:26,640 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/85e4656b645441dbbe74ea29904f5eb5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/85e4656b645441dbbe74ea29904f5eb5 2023-04-27 01:55:26,645 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/85e4656b645441dbbe74ea29904f5eb5, entries=100, sequenceid=164, filesize=9.4 K 2023-04-27 01:55:26,646 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/04fa7ee4836c4de288854fa655276e77 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/04fa7ee4836c4de288854fa655276e77 2023-04-27 01:55:26,650 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/04fa7ee4836c4de288854fa655276e77, entries=100, sequenceid=164, filesize=9.4 K 2023-04-27 01:55:26,651 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for b5f127509ba53373491a414d00be9fa1 in 518ms, sequenceid=164, compaction requested=true 2023-04-27 01:55:26,651 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:26,651 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:26,653 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,654 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,654 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c972fc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:26,657 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:26,657 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,659 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:26,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:26,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:26,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:26,661 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:26,661 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,661 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:26,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:26,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:26,664 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:26,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:26,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:26,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:26,664 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:26,665 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,665 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,665 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,666 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,667 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,670 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,670 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,670 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,670 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:26,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:26,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:26,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:26,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:26,670 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,670 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:26,670 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:26,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,770 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:26,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:26,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:26,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:26,776 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:26,777 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,777 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,780 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:26,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:26,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:26,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:26,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:26,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:26,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:26,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:26,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:26,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:26,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:26,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:26,784 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d26c0b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:26,786 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:26,786 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:26,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:26,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:26,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:26,794 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:26,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:26,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:26,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:26,794 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:26,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:26,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:26,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:26,803 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:26,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:26,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,803 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:26,803 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,803 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,803 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,803 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,803 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,903 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:26,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:26,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:26,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:26,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:26,908 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:26,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:26,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:26,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:26,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,909 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,910 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:26,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,911 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:26,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:26,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:26,912 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:26,912 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:26,912 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:26,912 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:26,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ff07f4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:26,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:26,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,922 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:26,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,923 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:26,926 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:26,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:26,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:26,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:26,927 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:26,928 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,928 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,928 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:26,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:26,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:26,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:26,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:26,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:26,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:26,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:26,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:26,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:26,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:26,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:26,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:26,934 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:26,934 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:26,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:26,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:26,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:26,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:26,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:26,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:26,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:26,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:26,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:27,034 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,038 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,041 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,041 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,041 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,041 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,044 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,044 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,044 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:27,044 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,046 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e22ebc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:27,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:27,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,049 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:27,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,057 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:27,059 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:27,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:27,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:27,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:27,059 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:27,062 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,062 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,062 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,064 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:27,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,066 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,066 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:27,066 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:27,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:27,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560527074, max timestamp=1682560527103Num uniques 859; 2023-04-27 01:55:27,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:27,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560527074, max timestamp=1682560527103Num uniques 859; 2023-04-27 01:55:27,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:27,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560527074, max timestamp=1682560527103Num uniques 859; 2023-04-27 01:55:27,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:27,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560587148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560587151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560587152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,166 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,170 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,171 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,171 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,172 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,172 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,172 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:27,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:27,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:27,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6993fa0e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:27,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,180 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:27,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:27,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:27,184 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:27,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:27,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:27,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:27,184 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:27,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,186 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,188 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,191 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:27,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:27,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,192 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:27,192 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:27,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:27,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560587260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560587260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560587263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,292 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,295 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:27,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:27,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bd36103[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:27,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:27,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,305 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:27,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:27,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:27,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:27,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:27,313 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:27,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:27,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:27,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:27,313 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:27,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,318 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:27,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,318 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:27,318 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:27,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,408 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427eb5d8a51ec5a453e90da146aee40aa6d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eb5d8a51ec5a453e90da146aee40aa6d 2023-04-27 01:55:27,413 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=176, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/b55ef3d3ca65419980f63f391646aaaa 2023-04-27 01:55:27,418 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,425 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,425 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,425 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,425 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/32f5705e72ee435297ee48749b8f1d5a 2023-04-27 01:55:27,428 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,430 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,430 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,430 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:27,430 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:27,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:27,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:27,431 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:27,433 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7179052a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:27,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:27,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:27,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:27,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:27,441 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:27,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:27,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:27,442 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:27,442 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:27,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:27,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:27,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:27,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,449 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,449 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,448 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,449 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,449 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:27,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,449 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:27,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560587470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560587470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560587471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,549 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,552 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,552 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,552 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,552 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,554 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,556 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,556 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,556 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:27,556 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:27,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26395277[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:27,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:27,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,560 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:27,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:27,562 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:27,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:27,565 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:27,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:27,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:27,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:27,565 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:27,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,567 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:27,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:27,571 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:27,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,571 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:27,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,671 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,674 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,674 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,678 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,678 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,678 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:27,678 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:27,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:27,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500c8f34[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:27,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:27,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,682 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,683 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:27,683 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:27,683 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:27,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:27,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:27,687 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:27,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:27,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:27,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:27,687 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:27,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,689 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,695 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:27,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:27,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,696 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:27,696 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:27,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:27,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560587779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560587780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:27,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560587780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,796 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,799 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,799 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,804 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,804 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,804 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:27,805 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:27,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,806 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5057374f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:27,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:27,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:27,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:27,810 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:27,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:27,814 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:27,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:27,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:27,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:27,814 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:27,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:27,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:27,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:27,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:27,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:27,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:27,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:27,821 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:27,821 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:27,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:27,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/45e27d1a56a349d88d3824283433cab1 2023-04-27 01:55:27,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/b55ef3d3ca65419980f63f391646aaaa as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b55ef3d3ca65419980f63f391646aaaa 2023-04-27 01:55:27,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b55ef3d3ca65419980f63f391646aaaa, entries=200, sequenceid=176, filesize=32.1 K 2023-04-27 01:55:27,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/32f5705e72ee435297ee48749b8f1d5a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/32f5705e72ee435297ee48749b8f1d5a 2023-04-27 01:55:27,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/32f5705e72ee435297ee48749b8f1d5a, entries=100, sequenceid=176, filesize=9.4 K 2023-04-27 01:55:27,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/45e27d1a56a349d88d3824283433cab1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/45e27d1a56a349d88d3824283433cab1 2023-04-27 01:55:27,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/45e27d1a56a349d88d3824283433cab1, entries=100, sequenceid=176, filesize=9.4 K 2023-04-27 01:55:27,854 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b5f127509ba53373491a414d00be9fa1 in 864ms, sequenceid=176, compaction requested=true 2023-04-27 01:55:27,854 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:27,854 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:27,854 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:27,855 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:27,854 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:27,855 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:27,857 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155136 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:27,857 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67200 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:27,857 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/A is initiating minor compaction (all files) 2023-04-27 01:55:27,857 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/B is initiating minor compaction (all files) 2023-04-27 01:55:27,857 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/A in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,857 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/B in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,857 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b8946eafdfdf48acb509eae9828d950a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d502d235b57342a6b51d5bcf54961b35, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/8cd75142a24b4ee4bce2e1119bdd9027, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7a0843ed25594282bfbe4b83c1921d19, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/c2592bbbc225433c9ab663291a8e83f3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b55ef3d3ca65419980f63f391646aaaa] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=151.5 K 2023-04-27 01:55:27,857 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/afb242fc39f842ec87b51d1194fbb246, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/48219db5b6a248e6875abab624b67838, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ab0be9be2784140b681bf03d4ed25f9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ef76e5d90b741eaacf273a47b9b5914, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/85e4656b645441dbbe74ea29904f5eb5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/32f5705e72ee435297ee48749b8f1d5a] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=65.6 K 2023-04-27 01:55:27,858 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b8946eafdfdf48acb509eae9828d950a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682560521146 2023-04-27 01:55:27,858 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting afb242fc39f842ec87b51d1194fbb246, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682560521146 2023-04-27 01:55:27,858 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d502d235b57342a6b51d5bcf54961b35, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682560521813 2023-04-27 01:55:27,858 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 48219db5b6a248e6875abab624b67838, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682560521813 2023-04-27 01:55:27,858 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8cd75142a24b4ee4bce2e1119bdd9027, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682560522959 2023-04-27 01:55:27,858 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9ab0be9be2784140b681bf03d4ed25f9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682560522959 2023-04-27 01:55:27,859 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 7a0843ed25594282bfbe4b83c1921d19, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682560524116 2023-04-27 01:55:27,859 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9ef76e5d90b741eaacf273a47b9b5914, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682560524116 2023-04-27 01:55:27,859 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c2592bbbc225433c9ab663291a8e83f3, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682560524746 2023-04-27 01:55:27,859 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 85e4656b645441dbbe74ea29904f5eb5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682560524746 2023-04-27 01:55:27,859 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b55ef3d3ca65419980f63f391646aaaa, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682560526970 2023-04-27 01:55:27,859 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 32f5705e72ee435297ee48749b8f1d5a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682560526971 2023-04-27 01:55:27,872 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#B#compaction#324 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:27,888 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#A#compaction#325 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:27,921 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:27,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:27,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:27,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:27,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:27,925 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:27,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:27,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:27,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,928 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:27,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:27,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:27,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:27,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:27,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:27,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:27,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:27,932 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:27,932 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:27,932 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 01:55:27,932 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:27,932 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:27,932 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:27,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:27,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:27,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:27,948 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042796feec413f0c403080edb270dc3f120a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042796feec413f0c403080edb270dc3f120a 2023-04-27 01:55:28,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:28,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:28,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560588225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560588229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,290 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/11a2cbe177c548619b88719118b86e12 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/11a2cbe177c548619b88719118b86e12 2023-04-27 01:55:28,293 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b5f127509ba53373491a414d00be9fa1/B of b5f127509ba53373491a414d00be9fa1 into 11a2cbe177c548619b88719118b86e12(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:28,294 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:28,294 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/B, priority=10, startTime=1682560527854; duration=0sec 2023-04-27 01:55:28,294 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:28,294 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:28,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560588288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560588289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560588289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,296 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67200 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:28,296 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/C is initiating minor compaction (all files) 2023-04-27 01:55:28,296 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/C in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:28,297 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/698aa6533a7b4aa9b0fb0d56dfa8e16a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/09bd3b4b91d04befaf89d9c7a897b920, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d881e1c0f1d74093a79249f726db49be, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/25de986d2ea545b5b9a9ff6c9e21a3ca, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/04fa7ee4836c4de288854fa655276e77, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/45e27d1a56a349d88d3824283433cab1] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=65.6 K 2023-04-27 01:55:28,297 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 698aa6533a7b4aa9b0fb0d56dfa8e16a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682560521146 2023-04-27 01:55:28,297 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 09bd3b4b91d04befaf89d9c7a897b920, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682560521813 2023-04-27 01:55:28,298 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d881e1c0f1d74093a79249f726db49be, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682560522959 2023-04-27 01:55:28,298 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 25de986d2ea545b5b9a9ff6c9e21a3ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682560524116 2023-04-27 01:55:28,299 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 04fa7ee4836c4de288854fa655276e77, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682560524746 2023-04-27 01:55:28,299 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 45e27d1a56a349d88d3824283433cab1, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682560526971 2023-04-27 01:55:28,307 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/ce26187c80fb4450b0d333398bf19073 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ce26187c80fb4450b0d333398bf19073 2023-04-27 01:55:28,311 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b5f127509ba53373491a414d00be9fa1/A of b5f127509ba53373491a414d00be9fa1 into ce26187c80fb4450b0d333398bf19073(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:28,311 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:28,311 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/A, priority=10, startTime=1682560527854; duration=0sec 2023-04-27 01:55:28,312 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:28,312 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#C#compaction#327 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:28,321 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/9bcba7fea6934ac3b15a3c9973dd1374 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9bcba7fea6934ac3b15a3c9973dd1374 2023-04-27 01:55:28,324 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b5f127509ba53373491a414d00be9fa1/C of b5f127509ba53373491a414d00be9fa1 into 9bcba7fea6934ac3b15a3c9973dd1374(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:28,324 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:28,324 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/C, priority=10, startTime=1682560527855; duration=0sec 2023-04-27 01:55:28,325 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:28,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560588332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560588336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,354 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=212, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/45786177c08848aaa2d1633e9aff386a 2023-04-27 01:55:28,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560588533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560588539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,763 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4 2023-04-27 01:55:28,772 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/d1e95477baf7457e8eb8c3d59d81b43b 2023-04-27 01:55:28,776 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/45786177c08848aaa2d1633e9aff386a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/45786177c08848aaa2d1633e9aff386a 2023-04-27 01:55:28,781 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/45786177c08848aaa2d1633e9aff386a, entries=150, sequenceid=212, filesize=25.3 K 2023-04-27 01:55:28,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4 2023-04-27 01:55:28,787 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4, entries=150, sequenceid=212, filesize=11.8 K 2023-04-27 01:55:28,788 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/d1e95477baf7457e8eb8c3d59d81b43b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d1e95477baf7457e8eb8c3d59d81b43b 2023-04-27 01:55:28,792 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d1e95477baf7457e8eb8c3d59d81b43b, entries=150, sequenceid=212, filesize=11.8 K 2023-04-27 01:55:28,793 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for b5f127509ba53373491a414d00be9fa1 in 861ms, sequenceid=212, compaction requested=false 2023-04-27 01:55:28,793 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:28,793 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:28,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:28,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:28,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:28,793 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:28,795 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:28,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:28,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69be383a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:28,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:28,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,800 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:28,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:28,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:28,811 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:28,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:28,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:28,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:28,811 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:28,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,813 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:28,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:28,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:28,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:28,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:28,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:28,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:28,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,823 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:28,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:28,824 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:28,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:28,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:55:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:28,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560588890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560528879, max timestamp=1682560528895Num uniques 859; 2023-04-27 01:55:28,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560528879, max timestamp=1682560528895Num uniques 859; 2023-04-27 01:55:28,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:55:28,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:55:28,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560528879, max timestamp=1682560528895Num uniques 859; 2023-04-27 01:55:28,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:55:28,924 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:28,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:28,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:28,927 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:28,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:28,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:28,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:28,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:28,931 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:28,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:28,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:28,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:28,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:28,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:28,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:28,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:28,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:28,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:28,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:28,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5786590[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:28,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:28,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:28,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:28,938 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:28,941 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:28,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:28,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:28,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:28,941 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:28,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:28,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:28,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:28,945 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,946 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:28,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:28,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:28,951 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:28,951 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:28,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:28,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560588986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560588998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,051 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:29,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:29,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:29,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:29,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:29,054 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:29,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:29,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:29,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:29,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:29,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:29,058 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,058 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:29,058 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:29,058 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:29,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:29,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31b92f24[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:29,062 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:29,062 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,062 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:29,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:29,066 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:29,066 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:29,066 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:29,066 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:29,066 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:29,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,072 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:29,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:29,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,072 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:29,072 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:29,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560589093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,172 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:29,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:29,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:29,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:29,176 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:29,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:29,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:29,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,178 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:29,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:29,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:29,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:29,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:29,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58bd8b13[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:29,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:29,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,184 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:29,187 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:29,187 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,187 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:29,189 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:29,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:29,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:29,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:29,189 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:29,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,191 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,194 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:29,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:29,194 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:29,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,194 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:29,194 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:29,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560589203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,270 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427fe9ae382cef846199b7d72e66ed8e86c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fe9ae382cef846199b7d72e66ed8e86c 2023-04-27 01:55:29,275 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=226, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/3ddf6d59437d468e9ceddc81080d9fc5 2023-04-27 01:55:29,294 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:29,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:29,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:29,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:29,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:29,298 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:29,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:29,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:29,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560589297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:29,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:29,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:29,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:29,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:29,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:29,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560589298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,304 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560589299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:29,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40da07d6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:29,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:29,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,307 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:29,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560589301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,309 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:29,311 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:29,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:29,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:29,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:29,311 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:29,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:29,319 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:29,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,319 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:29,319 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:29,420 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:29,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:29,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:29,423 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:29,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:29,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:29,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,428 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:29,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:29,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:29,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:29,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:29,442 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c8aeb8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:29,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:29,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:29,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,450 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,450 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:29,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:29,454 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:29,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:29,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:29,454 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:29,454 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:29,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,456 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,456 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,456 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,458 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:29,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,462 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,462 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,462 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,462 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:29,462 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:29,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560589512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,562 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:29,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:29,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:29,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:29,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:29,568 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:29,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:29,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:29,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,571 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:29,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,576 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:29,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,584 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:29,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:29,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:29,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:29,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,588 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,588 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,588 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef0714a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:29,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:29,591 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,593 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,593 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:29,593 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:29,593 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:29,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:29,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:29,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,594 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,594 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:29,597 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:29,597 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:29,597 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:29,597 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:29,597 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:29,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:29,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,606 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,609 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:29,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:29,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:29,609 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,609 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,610 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,610 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:29,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:29,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:29,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:29,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:29,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560589613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/cc0eba73b9834913aaf46a905762a2d0 2023-04-27 01:55:29,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c 2023-04-27 01:55:29,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/3ddf6d59437d468e9ceddc81080d9fc5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3ddf6d59437d468e9ceddc81080d9fc5 2023-04-27 01:55:29,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3ddf6d59437d468e9ceddc81080d9fc5, entries=200, sequenceid=226, filesize=32.1 K 2023-04-27 01:55:29,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/cc0eba73b9834913aaf46a905762a2d0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/cc0eba73b9834913aaf46a905762a2d0 2023-04-27 01:55:29,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/cc0eba73b9834913aaf46a905762a2d0, entries=150, sequenceid=226, filesize=11.8 K 2023-04-27 01:55:29,710 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:29,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c 2023-04-27 01:55:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:29,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:29,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c, entries=150, sequenceid=226, filesize=11.8 K 2023-04-27 01:55:29,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for b5f127509ba53373491a414d00be9fa1 in 874ms, sequenceid=226, compaction requested=true 2023-04-27 01:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:29,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:29,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:29,715 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:29,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:29,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:29,715 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:29,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:29,715 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:29,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:29,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:29,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,716 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:29,716 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:29,716 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/B is initiating minor compaction (all files) 2023-04-27 01:55:29,716 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/A is initiating minor compaction (all files) 2023-04-27 01:55:29,716 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/A in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,717 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/B in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,717 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ce26187c80fb4450b0d333398bf19073, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/45786177c08848aaa2d1633e9aff386a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3ddf6d59437d468e9ceddc81080d9fc5] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=83.1 K 2023-04-27 01:55:29,717 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/11a2cbe177c548619b88719118b86e12, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/cc0eba73b9834913aaf46a905762a2d0] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=35.6 K 2023-04-27 01:55:29,717 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:29,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:29,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:29,717 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ce26187c80fb4450b0d333398bf19073, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682560524752 2023-04-27 01:55:29,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:29,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:29,717 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 11a2cbe177c548619b88719118b86e12, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682560524752 2023-04-27 01:55:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:29,718 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 45786177c08848aaa2d1633e9aff386a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1682560527130 2023-04-27 01:55:29,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:29,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:29,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:29,718 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 0bbdd5fafdd14eb9ad32ec4dfb81d5d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1682560527130 2023-04-27 01:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:29,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:29,719 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cc0eba73b9834913aaf46a905762a2d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682560528220 2023-04-27 01:55:29,719 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:29,719 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:29,719 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3ddf6d59437d468e9ceddc81080d9fc5, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682560528220 2023-04-27 01:55:29,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:29,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:29,719 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-27 01:55:29,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:29,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:29,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:29,720 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:29,720 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:29,720 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:29,729 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#B#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:29,742 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#A#compaction#334 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:30,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:30,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:30,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560590088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:30,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560590116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,152 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/22016fd6bf01444883623580ccd8ea5c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/22016fd6bf01444883623580ccd8ea5c 2023-04-27 01:55:30,158 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/B of b5f127509ba53373491a414d00be9fa1 into 22016fd6bf01444883623580ccd8ea5c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:30,158 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:30,158 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/B, priority=13, startTime=1682560529715; duration=0sec 2023-04-27 01:55:30,158 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:30,158 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:30,159 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:30,159 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/C is initiating minor compaction (all files) 2023-04-27 01:55:30,159 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/C in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:30,160 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9bcba7fea6934ac3b15a3c9973dd1374, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d1e95477baf7457e8eb8c3d59d81b43b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=35.6 K 2023-04-27 01:55:30,160 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9bcba7fea6934ac3b15a3c9973dd1374, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682560524752 2023-04-27 01:55:30,160 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d1e95477baf7457e8eb8c3d59d81b43b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1682560527130 2023-04-27 01:55:30,161 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8ae8ca9a0c9449dbaa8aa23ff3143e1c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682560528220 2023-04-27 01:55:30,164 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042731c03a6d51fe47bf86e9e6c43e7aa405 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042731c03a6d51fe47bf86e9e6c43e7aa405 2023-04-27 01:55:30,169 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#C#compaction#336 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:30,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:30,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560590200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:30,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560590406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,581 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/eaa2932d63794c86a53b0814cecf2d35 2023-04-27 01:55:30,601 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/ccacc4bf5d9f4aeba493c5ec681d1019 2023-04-27 01:55:30,605 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/97be6057afd44906b576f0ba1df5cb88 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97be6057afd44906b576f0ba1df5cb88 2023-04-27 01:55:30,613 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/C of b5f127509ba53373491a414d00be9fa1 into 97be6057afd44906b576f0ba1df5cb88(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:30,614 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:30,614 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/C, priority=13, startTime=1682560529715; duration=0sec 2023-04-27 01:55:30,614 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:30,617 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/3cc4248549eb44fe86653c3e8b1acab9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3cc4248549eb44fe86653c3e8b1acab9 2023-04-27 01:55:30,619 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/15445a5df5b64737a733ffb346f209b2 2023-04-27 01:55:30,623 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/eaa2932d63794c86a53b0814cecf2d35 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/eaa2932d63794c86a53b0814cecf2d35 2023-04-27 01:55:30,624 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/A of b5f127509ba53373491a414d00be9fa1 into 3cc4248549eb44fe86653c3e8b1acab9(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:30,624 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:30,624 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/A, priority=13, startTime=1682560529715; duration=0sec 2023-04-27 01:55:30,624 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:30,627 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/eaa2932d63794c86a53b0814cecf2d35, entries=150, sequenceid=262, filesize=25.4 K 2023-04-27 01:55:30,628 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/ccacc4bf5d9f4aeba493c5ec681d1019 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/ccacc4bf5d9f4aeba493c5ec681d1019 2023-04-27 01:55:30,632 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/ccacc4bf5d9f4aeba493c5ec681d1019, entries=150, sequenceid=262, filesize=11.9 K 2023-04-27 01:55:30,632 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/15445a5df5b64737a733ffb346f209b2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/15445a5df5b64737a733ffb346f209b2 2023-04-27 01:55:30,637 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/15445a5df5b64737a733ffb346f209b2, entries=150, sequenceid=262, filesize=11.9 K 2023-04-27 01:55:30,638 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for b5f127509ba53373491a414d00be9fa1 in 919ms, sequenceid=262, compaction requested=false 2023-04-27 01:55:30,638 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:30,638 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:30,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:30,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:30,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,640 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:30,640 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:30,642 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,642 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,642 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:30,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a218894[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:30,645 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:30,645 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,647 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:30,650 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:30,650 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:30,650 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:30,650 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,650 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:30,650 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:30,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:30,654 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:30,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:30,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:30,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:30,655 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:30,656 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,656 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:30,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,656 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,657 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,658 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,660 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:30,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:30,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:30,661 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:30,661 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:30,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:30,662 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:30,663 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:30,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:30,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:30,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:30,722 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:30,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:30,761 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:30,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:30,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:30,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:30,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:30,764 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:30,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:30,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:30,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:30,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,766 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:30,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:30,767 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,767 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:30,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:30,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:30,768 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:30,768 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:30,768 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:30,768 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:30,772 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,772 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,773 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,773 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:30,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad5d2b1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:30,777 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:30,778 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,779 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,779 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:30,779 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:30,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:30,779 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:30,781 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:30,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:30,783 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:30,783 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:30,783 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:30,783 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:30,783 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:30,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:30,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,787 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:30,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:30,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:30,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:30,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:30,789 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:30,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:30,789 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:30,789 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:30,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,789 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:30,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:30,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560590816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,889 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:30,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:30,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:30,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:30,892 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:30,892 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:30,892 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:30,892 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:30,893 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,893 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,894 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:30,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:30,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:30,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:30,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:30,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:30,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:30,897 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:30,897 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:30,897 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:30,897 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:30,899 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,899 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:30,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ab0a3d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:30,901 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:30,901 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,902 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:30,902 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:30,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:30,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:30,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:30,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:30,906 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:30,906 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:30,906 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:30,906 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:30,906 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:30,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:30,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:30,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:30,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:30,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:30,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:30,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,914 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:30,914 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,914 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:30,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:30,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560590922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,014 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,018 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,018 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,018 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,018 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,018 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,018 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,020 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,020 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:31,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,021 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,021 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,021 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,022 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@642c4263[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,025 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,025 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,026 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,026 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,032 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,032 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,032 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,032 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,032 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,033 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,033 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,034 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,034 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,035 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,047 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,047 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,047 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,047 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,048 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,048 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,048 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,048 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,048 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,050 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,050 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:31,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560591127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560591132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,140 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304276b6564a1e7a14095a96e0ed5799dacc0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276b6564a1e7a14095a96e0ed5799dacc0 2023-04-27 01:55:31,150 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,153 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,153 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,153 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,153 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,154 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,154 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:31,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,157 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,157 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,157 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,157 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,159 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,159 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,160 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,160 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220f035a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,162 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,162 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,162 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,162 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,162 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,162 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,166 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,166 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,166 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,166 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,169 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,169 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,169 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,169 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,169 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:31,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,176 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,176 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,276 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,280 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,282 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,282 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,282 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:31,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,284 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,284 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,284 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,284 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,285 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,285 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,286 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,286 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bb1df0[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,291 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,291 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,293 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,293 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,295 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,295 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,295 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,299 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:31,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,299 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:31,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40294 deadline: 1682560591301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,307 DEBUG [Thread-2178] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:31,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:31,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560591314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,319 DEBUG [Thread-2172] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:31,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:31,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560591318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,321 DEBUG [Thread-2174] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:31,399 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,404 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,404 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,404 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,404 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,404 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,405 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,407 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,407 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,413 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,413 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:31,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,414 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,415 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,415 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,416 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,416 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,416 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,416 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,418 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,418 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,418 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,418 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17801ff8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,422 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,422 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,422 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,422 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,427 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,427 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,427 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,427 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,427 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,429 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,429 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,429 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,436 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,436 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:31,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560591440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,540 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,543 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,544 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,544 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,545 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,546 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/f876eeecc7a1417093a0d8b7b1c94f72 2023-04-27 01:55:31,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,546 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,547 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,547 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,547 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,548 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,548 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,548 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,548 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,549 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,549 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,551 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11b97831[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,551 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,554 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,554 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,556 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,561 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,562 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,562 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,563 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,563 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,563 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,566 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,578 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,578 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,578 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,678 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,682 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:31,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,687 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf6bc8f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,694 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,694 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,698 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:31,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,701 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,701 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,701 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,702 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,702 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,802 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,805 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,805 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,805 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,805 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,806 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,806 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:31,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,809 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,809 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,809 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,809 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,811 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ac7c9c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,816 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,820 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,820 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,821 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,822 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,826 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:31,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,826 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,826 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,827 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,827 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,927 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:31,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:31,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:31,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:31,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:31,930 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:31,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:31,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:31,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,931 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,931 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:31,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:31,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:31,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:31,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:31,933 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:31,934 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52c4d40e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:31,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:31,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:31,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:31,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:31,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,938 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:31,944 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:31,946 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:31,946 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:31,946 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:31,946 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:31,946 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:31,948 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:31,948 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:31,948 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,948 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:31,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40264 deadline: 1682560591950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,955 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:31,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:31,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:31,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:31,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:31,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:31,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:31,956 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:31,956 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:31,966 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/42318b871d144fa6a571d7539c286ed4 2023-04-27 01:55:32,057 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:32,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:32,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:32,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:32,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:32,060 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:32,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:32,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:32,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:32,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:32,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:32,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:32,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:32,065 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:32,065 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:32,065 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,065 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:32,065 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:32,065 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:32,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:32,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:32,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:32,067 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:32,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8665697[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:32,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:32,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,070 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:32,070 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:32,070 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:32,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:32,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:32,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:32,074 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:32,074 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:32,074 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:32,074 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:32,074 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:32,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,077 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:32,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:32,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:32,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,080 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:32,080 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:32,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:32,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,081 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,081 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,081 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:32,181 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:32,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:32,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:32,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:32,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:32,185 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:32,185 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,185 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:32,185 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:32,186 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,186 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:32,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:32,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:32,189 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,189 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:32,189 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:32,189 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:32,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:32,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768a91d0[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:32,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:32,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,195 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,195 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,195 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:32,195 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:32,195 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:32,197 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,197 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:32,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:32,199 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:32,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:32,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:32,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:32,199 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:32,201 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,201 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,201 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,201 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,202 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,206 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:32,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:32,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,206 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,207 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,207 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,207 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:32,207 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:32,307 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:32,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:32,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:32,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:32,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:32,310 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:32,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:32,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:32,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,312 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:32,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:32,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:32,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:32,314 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,314 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:32,314 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:32,314 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:32,315 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50e44b01[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:32,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:32,317 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:32,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:32,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:32,319 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,319 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:32,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:32,321 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,321 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:32,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:32,323 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:32,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:32,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:32,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:32,323 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:32,324 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,324 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,324 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:32,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:32,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:32,328 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:32,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:32,328 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:32,328 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:32,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/c7a5e0ebd0cd4b66818c924f82e4ab8f 2023-04-27 01:55:32,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/f876eeecc7a1417093a0d8b7b1c94f72 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f876eeecc7a1417093a0d8b7b1c94f72 2023-04-27 01:55:32,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f876eeecc7a1417093a0d8b7b1c94f72, entries=200, sequenceid=277, filesize=32.3 K 2023-04-27 01:55:32,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/42318b871d144fa6a571d7539c286ed4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/42318b871d144fa6a571d7539c286ed4 2023-04-27 01:55:32,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/42318b871d144fa6a571d7539c286ed4, entries=150, sequenceid=277, filesize=11.9 K 2023-04-27 01:55:32,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/c7a5e0ebd0cd4b66818c924f82e4ab8f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c7a5e0ebd0cd4b66818c924f82e4ab8f 2023-04-27 01:55:32,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c7a5e0ebd0cd4b66818c924f82e4ab8f, entries=150, sequenceid=277, filesize=11.9 K 2023-04-27 01:55:32,399 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b5f127509ba53373491a414d00be9fa1 in 1677ms, sequenceid=277, compaction requested=true 2023-04-27 01:55:32,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:32,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:32,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:32,399 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:32,399 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:32,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:32,400 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85506 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:32,400 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/A is initiating minor compaction (all files) 2023-04-27 01:55:32,400 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:32,400 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/A in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,400 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/B is initiating minor compaction (all files) 2023-04-27 01:55:32,400 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3cc4248549eb44fe86653c3e8b1acab9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/eaa2932d63794c86a53b0814cecf2d35, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f876eeecc7a1417093a0d8b7b1c94f72] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=83.5 K 2023-04-27 01:55:32,400 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/B in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,400 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/22016fd6bf01444883623580ccd8ea5c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/ccacc4bf5d9f4aeba493c5ec681d1019, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/42318b871d144fa6a571d7539c286ed4] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=36.0 K 2023-04-27 01:55:32,401 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 22016fd6bf01444883623580ccd8ea5c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682560528220 2023-04-27 01:55:32,401 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ccacc4bf5d9f4aeba493c5ec681d1019, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682560528967 2023-04-27 01:55:32,401 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 42318b871d144fa6a571d7539c286ed4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560530063 2023-04-27 01:55:32,403 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3cc4248549eb44fe86653c3e8b1acab9, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682560528220 2023-04-27 01:55:32,404 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting eaa2932d63794c86a53b0814cecf2d35, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682560528967 2023-04-27 01:55:32,404 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f876eeecc7a1417093a0d8b7b1c94f72, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560530063 2023-04-27 01:55:32,409 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#B#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:32,428 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:32,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:32,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:32,433 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#A#compaction#343 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:32,433 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:32,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:32,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:32,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,437 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/186e055749624dfaa1998ae46d70b540 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/186e055749624dfaa1998ae46d70b540 2023-04-27 01:55:32,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:32,443 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/B of b5f127509ba53373491a414d00be9fa1 into 186e055749624dfaa1998ae46d70b540(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:32,443 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:32,443 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/B, priority=13, startTime=1682560532399; duration=0sec 2023-04-27 01:55:32,443 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:32,443 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:32,444 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:32,444 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/C is initiating minor compaction (all files) 2023-04-27 01:55:32,444 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/C in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,444 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97be6057afd44906b576f0ba1df5cb88, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/15445a5df5b64737a733ffb346f209b2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c7a5e0ebd0cd4b66818c924f82e4ab8f] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=36.0 K 2023-04-27 01:55:32,445 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 97be6057afd44906b576f0ba1df5cb88, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682560528220 2023-04-27 01:55:32,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:32,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:32,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:32,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:32,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:32,445 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 15445a5df5b64737a733ffb346f209b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682560528967 2023-04-27 01:55:32,445 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c7a5e0ebd0cd4b66818c924f82e4ab8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560530063 2023-04-27 01:55:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:32,448 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:32,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:32,452 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#C#compaction#344 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:32,472 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/1a012514a9fa4071b3ed16576fb62e9d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/1a012514a9fa4071b3ed16576fb62e9d 2023-04-27 01:55:32,477 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/C of b5f127509ba53373491a414d00be9fa1 into 1a012514a9fa4071b3ed16576fb62e9d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:32,477 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042780dc4b2c088544ceb6dd24bd465a9785 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042780dc4b2c088544ceb6dd24bd465a9785 2023-04-27 01:55:32,477 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:32,477 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/C, priority=13, startTime=1682560532399; duration=0sec 2023-04-27 01:55:32,477 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:32,484 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/b669d1b87e6a44a4ad290ae457b7b804 2023-04-27 01:55:32,909 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/531df00aa6794cd2927e9a998ce3e8fa 2023-04-27 01:55:32,924 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/97332050e5ec4e138d2ef5a8d5158bf0 2023-04-27 01:55:32,930 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/b669d1b87e6a44a4ad290ae457b7b804 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b669d1b87e6a44a4ad290ae457b7b804 2023-04-27 01:55:32,933 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b669d1b87e6a44a4ad290ae457b7b804, entries=150, sequenceid=298, filesize=25.4 K 2023-04-27 01:55:32,934 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/531df00aa6794cd2927e9a998ce3e8fa as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/531df00aa6794cd2927e9a998ce3e8fa 2023-04-27 01:55:32,949 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/531df00aa6794cd2927e9a998ce3e8fa, entries=150, sequenceid=298, filesize=11.9 K 2023-04-27 01:55:32,950 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/97332050e5ec4e138d2ef5a8d5158bf0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97332050e5ec4e138d2ef5a8d5158bf0 2023-04-27 01:55:32,954 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97332050e5ec4e138d2ef5a8d5158bf0, entries=150, sequenceid=298, filesize=11.9 K 2023-04-27 01:55:32,955 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for b5f127509ba53373491a414d00be9fa1 in 507ms, sequenceid=298, compaction requested=false 2023-04-27 01:55:32,955 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:32,955 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:32,956 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f481b7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:32,958 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:32,958 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:32,960 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,960 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:32,961 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:32,964 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:32,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:32,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:32,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:32,964 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:32,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,968 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,969 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:32,969 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,970 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,972 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,972 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,972 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:32,972 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:32,972 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,972 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:32,973 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:32,972 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:32,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:32,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:32,972 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:32,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:32,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:32,973 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:32,973 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:32,973 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:32,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,073 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:33,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:33,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:33,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:33,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:33,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:33,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:33,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:33,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:33,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:33,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:33,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:33,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:33,077 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:33,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:33,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:33,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,078 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:33,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:33,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:33,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:33,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:33,088 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,088 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:33,088 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:33,088 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:33,090 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e9cb950[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:33,092 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:33,092 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,093 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427ea5a0fa91d0c4d5fbc999cac0ed63b2e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427ea5a0fa91d0c4d5fbc999cac0ed63b2e 2023-04-27 01:55:33,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:33,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:33,094 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:33,094 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,094 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:33,096 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,098 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/3e39559a7eea406aad6a7d2778b963dc 2023-04-27 01:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:33,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:33,102 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:33,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:33,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:33,102 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:33,102 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:33,106 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,107 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,108 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,111 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,111 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,111 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,111 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:33,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:33,111 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,111 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,111 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,111 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:33,111 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,111 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:33,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,112 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,112 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,112 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/b21eb3953a6b4bdea5508cd4089768d8 2023-04-27 01:55:33,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/36109ae633e1438395b7570d3b44f901 2023-04-27 01:55:33,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/3e39559a7eea406aad6a7d2778b963dc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3e39559a7eea406aad6a7d2778b963dc 2023-04-27 01:55:33,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3e39559a7eea406aad6a7d2778b963dc, entries=200, sequenceid=309, filesize=32.3 K 2023-04-27 01:55:33,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/b21eb3953a6b4bdea5508cd4089768d8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/b21eb3953a6b4bdea5508cd4089768d8 2023-04-27 01:55:33,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/b21eb3953a6b4bdea5508cd4089768d8, entries=150, sequenceid=309, filesize=11.9 K 2023-04-27 01:55:33,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/36109ae633e1438395b7570d3b44f901 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/36109ae633e1438395b7570d3b44f901 2023-04-27 01:55:33,149 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/36109ae633e1438395b7570d3b44f901, entries=150, sequenceid=309, filesize=11.9 K 2023-04-27 01:55:33,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=93.93 KB/96180 for b5f127509ba53373491a414d00be9fa1 in 76ms, sequenceid=309, compaction requested=true 2023-04-27 01:55:33,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:33,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:33,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 01:55:33,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-27 01:55:33,151 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-27 01:55:33,151 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 01:55:33,151 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 01:55:33,151 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. because compaction request was cancelled 2023-04-27 01:55:33,152 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:33,152 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:33,153 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/C is initiating minor compaction (all files) 2023-04-27 01:55:33,153 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/C in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,153 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/1a012514a9fa4071b3ed16576fb62e9d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97332050e5ec4e138d2ef5a8d5158bf0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/36109ae633e1438395b7570d3b44f901] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=36.3 K 2023-04-27 01:55:33,154 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1a012514a9fa4071b3ed16576fb62e9d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560530063 2023-04-27 01:55:33,154 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 97332050e5ec4e138d2ef5a8d5158bf0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682560530802 2023-04-27 01:55:33,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:33,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:55:33,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:33,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:33,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:33,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:33,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:33,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:33,156 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 36109ae633e1438395b7570d3b44f901, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682560533060 2023-04-27 01:55:33,169 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#C#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:33,208 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/08da19a445ec4978a1e828d7a7d3ce8a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/08da19a445ec4978a1e828d7a7d3ce8a 2023-04-27 01:55:33,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560593201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560593202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,211 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:33,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:33,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:33,214 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/C of b5f127509ba53373491a414d00be9fa1 into 08da19a445ec4978a1e828d7a7d3ce8a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:33,214 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:33,214 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/C, priority=13, startTime=1682560533150; duration=0sec 2023-04-27 01:55:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:33,214 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:33,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:33,215 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:33,215 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:33,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:33,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:33,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,216 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:33,216 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/B is initiating minor compaction (all files) 2023-04-27 01:55:33,216 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/B in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,216 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/186e055749624dfaa1998ae46d70b540, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/531df00aa6794cd2927e9a998ce3e8fa, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/b21eb3953a6b4bdea5508cd4089768d8] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=36.3 K 2023-04-27 01:55:33,216 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 186e055749624dfaa1998ae46d70b540, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560530063 2023-04-27 01:55:33,217 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 531df00aa6794cd2927e9a998ce3e8fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682560530802 2023-04-27 01:55:33,217 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b21eb3953a6b4bdea5508cd4089768d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682560533060 2023-04-27 01:55:33,218 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,218 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,220 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,220 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:33,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:33,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:33,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:33,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:33,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,232 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:33,234 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#B#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:33,235 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,235 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:33,235 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:33,235 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:33,236 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:33,236 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,236 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:33,236 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:33,238 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,238 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,238 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,238 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cbabf42[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:33,240 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:33,240 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,242 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:33,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:33,243 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:33,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,243 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,244 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:33,245 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,247 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/a5a7fffd8c664ff8a86aae1aeee0a921 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/a5a7fffd8c664ff8a86aae1aeee0a921 2023-04-27 01:55:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:33,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:33,249 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:33,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:33,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:33,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:33,250 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:33,251 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/B of b5f127509ba53373491a414d00be9fa1 into a5a7fffd8c664ff8a86aae1aeee0a921(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:33,251 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:33,251 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/B, priority=13, startTime=1682560533150; duration=0sec 2023-04-27 01:55:33,251 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:33,253 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,253 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,253 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,255 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,258 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,258 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:33,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:33,258 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:33,259 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:33,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,268 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/f55045c5438c4fe1a5b0a170edd22f40 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f55045c5438c4fe1a5b0a170edd22f40 2023-04-27 01:55:33,272 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b5f127509ba53373491a414d00be9fa1/A of b5f127509ba53373491a414d00be9fa1 into f55045c5438c4fe1a5b0a170edd22f40(size=26.0 K), total size for store is 83.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:33,272 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:33,272 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/A, priority=13, startTime=1682560532399; duration=0sec 2023-04-27 01:55:33,272 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:33,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560593311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560593311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,359 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:33,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:33,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:33,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:33,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:33,369 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:33,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:33,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:33,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,370 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,370 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,371 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,371 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:33,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:33,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:33,380 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:33,380 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,380 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:33,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,390 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:33,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e745797[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:33,392 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:33,392 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,393 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,394 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:33,395 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:33,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:33,398 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:33,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:33,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:33,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:33,398 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:33,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,402 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,405 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,405 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,405 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:33,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:33,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,405 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:33,405 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:33,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,506 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:33,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:33,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:33,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:33,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:33,512 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:33,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:33,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:33,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,515 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:33,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560593517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:33,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:33,520 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,520 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:33,520 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:33,521 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:33,521 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:33,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,523 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,523 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560593517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:33,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e0d8ed7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:33,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:33,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,527 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,527 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,527 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:33,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:33,531 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:33,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:33,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:33,536 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:33,536 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:33,536 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:33,536 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:33,536 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:33,538 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,538 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,538 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,539 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,545 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,549 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,549 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:33,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,549 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,549 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:33,549 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:33,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,601 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304274fb2cf4bdb0e448895e4c6c0f14cc339 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274fb2cf4bdb0e448895e4c6c0f14cc339 2023-04-27 01:55:33,607 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=327, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/fc1383b51b504c47bb3882f861c48198 2023-04-27 01:55:33,649 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:33,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:33,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:33,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:33,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:33,653 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:33,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:33,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:33,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,655 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:33,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:33,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:33,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:33,658 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:33,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:33,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:33,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:33,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:33,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,661 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,662 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,662 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:33,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3520e87f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:33,663 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:33,663 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,664 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:33,665 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,665 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:33,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:33,669 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:33,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:33,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:33,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:33,670 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:33,671 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,672 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:33,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:33,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,677 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:33,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,677 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:33,777 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:33,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:33,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:33,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:33,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:33,780 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:33,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:33,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:33,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,782 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,782 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,782 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,783 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:33,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:33,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:33,784 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,784 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:33,784 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:33,784 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:33,784 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:33,785 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:33,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c090136[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:33,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:33,789 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:33,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:33,790 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:33,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:33,798 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:33,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:33,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:33,801 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:33,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:33,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:33,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:33,801 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:33,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,803 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,806 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:33,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:33,806 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,807 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:33,807 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:33,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560593820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:33,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560593825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,907 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:33,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:33,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:33,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:33,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:33,912 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:33,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:33,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:33,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,913 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,914 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,914 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:33,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:33,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:33,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:33,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:33,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:33,916 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:33,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:33,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:33,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,920 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,921 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:33,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30ad2dcf[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:33,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:33,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,924 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:33,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:33,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:33,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:33,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:33,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,926 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:33,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:33,930 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:33,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:33,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:33,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:33,930 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:33,932 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,932 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,932 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:33,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:33,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:33,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:33,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:33,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:33,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:33,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:33,939 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:33,939 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:33,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:33,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:33,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:33,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:34,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/9e529de4342a4370bb28b26e29753278 2023-04-27 01:55:34,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/932c29acf459464a9d6e47b98f9ff396 2023-04-27 01:55:34,039 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:34,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:34,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:34,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:34,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:34,042 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:34,042 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:34,042 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:34,042 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:34,043 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,043 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,044 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,044 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:34,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:34,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:34,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/fc1383b51b504c47bb3882f861c48198 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/fc1383b51b504c47bb3882f861c48198 2023-04-27 01:55:34,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:34,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:34,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:34,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/fc1383b51b504c47bb3882f861c48198, entries=200, sequenceid=327, filesize=32.3 K 2023-04-27 01:55:34,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:34,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:34,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:34,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:34,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:34,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:34,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:34,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:34,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:34,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/9e529de4342a4370bb28b26e29753278 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9e529de4342a4370bb28b26e29753278 2023-04-27 01:55:34,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:34,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:34,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:34,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:34,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:34,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:34,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f4663b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:34,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:34,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:34,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:34,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:34,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:34,058 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:34,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:34,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:34,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:34,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:34,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9e529de4342a4370bb28b26e29753278, entries=150, sequenceid=327, filesize=11.9 K 2023-04-27 01:55:34,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:34,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/932c29acf459464a9d6e47b98f9ff396 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/932c29acf459464a9d6e47b98f9ff396 2023-04-27 01:55:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:34,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:34,063 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:34,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:34,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:34,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:34,063 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:34,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:34,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:34,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:34,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/932c29acf459464a9d6e47b98f9ff396, entries=150, sequenceid=327, filesize=11.9 K 2023-04-27 01:55:34,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b5f127509ba53373491a414d00be9fa1 in 913ms, sequenceid=327, compaction requested=true 2023-04-27 01:55:34,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:34,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:34,067 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:55:34,067 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-27 01:55:34,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:34,068 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,068 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 01:55:34,068 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 01:55:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:34,068 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. because compaction request was cancelled 2023-04-27 01:55:34,068 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-27 01:55:34,070 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118844 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:55:34,070 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/A is initiating minor compaction (all files) 2023-04-27 01:55:34,070 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/A in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:34,070 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f55045c5438c4fe1a5b0a170edd22f40, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b669d1b87e6a44a4ad290ae457b7b804, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3e39559a7eea406aad6a7d2778b963dc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/fc1383b51b504c47bb3882f861c48198] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=116.1 K 2023-04-27 01:55:34,070 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 01:55:34,070 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 01:55:34,070 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. because compaction request was cancelled 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:34,071 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f55045c5438c4fe1a5b0a170edd22f40, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682560530063 2023-04-27 01:55:34,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:34,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:34,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:34,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:34,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:34,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,071 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:34,072 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:34,072 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b669d1b87e6a44a4ad290ae457b7b804, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682560530802 2023-04-27 01:55:34,072 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3e39559a7eea406aad6a7d2778b963dc, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682560533060 2023-04-27 01:55:34,073 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting fc1383b51b504c47bb3882f861c48198, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682560533089 2023-04-27 01:55:34,086 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#A#compaction#356 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:34,172 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:34,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:34,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:34,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:34,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:34,176 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:34,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:34,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:34,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:34,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,179 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:34,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:34,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:34,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:34,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:34,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:34,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:34,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:34,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:34,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:34,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:34,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:34,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:34,181 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:34,181 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:55:34,181 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:34,181 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:34,181 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:34,181 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:34,181 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:34,181 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:34,193 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427c84ed25ef8fb4716b18440f345af3c97 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c84ed25ef8fb4716b18440f345af3c97 2023-04-27 01:55:34,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:34,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:34,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:34,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560594387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:34,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560594388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:34,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560594492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:34,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560594495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,598 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=344, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/9fc3185c715e4c428947f0ab225b6883 2023-04-27 01:55:34,609 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/1635d28a996f4e5194e6c27e498baf8b 2023-04-27 01:55:34,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:34,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560594700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:34,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:34,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560594701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560595004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560595008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,026 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/c5e5fc58618b4f37912400299e69edcb 2023-04-27 01:55:35,030 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/9fc3185c715e4c428947f0ab225b6883 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9fc3185c715e4c428947f0ab225b6883 2023-04-27 01:55:35,033 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9fc3185c715e4c428947f0ab225b6883, entries=150, sequenceid=344, filesize=25.4 K 2023-04-27 01:55:35,034 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/1635d28a996f4e5194e6c27e498baf8b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/1635d28a996f4e5194e6c27e498baf8b 2023-04-27 01:55:35,037 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/1635d28a996f4e5194e6c27e498baf8b, entries=150, sequenceid=344, filesize=11.9 K 2023-04-27 01:55:35,038 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/c5e5fc58618b4f37912400299e69edcb as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c5e5fc58618b4f37912400299e69edcb 2023-04-27 01:55:35,041 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c5e5fc58618b4f37912400299e69edcb, entries=150, sequenceid=344, filesize=11.9 K 2023-04-27 01:55:35,042 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b5f127509ba53373491a414d00be9fa1 in 861ms, sequenceid=344, compaction requested=true 2023-04-27 01:55:35,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:35,042 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:35,042 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:35,042 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:35,042 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,042 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:35,042 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:35,043 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a5170a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:35,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:35,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,047 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,047 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:35,047 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:35,047 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:35,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:35,049 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:35,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:35,051 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:35,051 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:35,051 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:35,051 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:35,051 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:35,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:35,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:35,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:35,056 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:35,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,062 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:35,062 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:35,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:35,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:35,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:35,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:35,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,063 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,063 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:35,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:35,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,163 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:35,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:35,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:35,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:35,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:35,167 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:35,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:35,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:35,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:35,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:35,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,168 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:35,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:35,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:35,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:35,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:35,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:35,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:35,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:35,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:35,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:35,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:35,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:35,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:35,173 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:55:35,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:35,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:35,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:35,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:35,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:35,174 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:35,305 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/f855732b98e04dafacc89e370fad95e1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f855732b98e04dafacc89e370fad95e1 2023-04-27 01:55:35,309 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b5f127509ba53373491a414d00be9fa1/A of b5f127509ba53373491a414d00be9fa1 into f855732b98e04dafacc89e370fad95e1(size=26.1 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 01:55:35,309 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:35,309 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/A, priority=12, startTime=1682560534067; duration=1sec 2023-04-27 01:55:35,309 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:35,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:35,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:35,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40336 deadline: 1682560595364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560595366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560595367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40306 deadline: 1682560595476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560595476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560595476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560595510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560595515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,597 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427c149d400bffb457fb7217ace76143fb6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c149d400bffb457fb7217ace76143fb6 2023-04-27 01:55:35,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560595680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560595680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560595681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560595985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:35,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:35,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560595988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560595992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,012 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/d5ade6ce7f5b42049d92ce2f3cbdb564 2023-04-27 01:55:36,032 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/41e79e35aab044eda59d0d2fe1a22020 2023-04-27 01:55:36,059 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/ec3ad296c9de43768556962b196b3447 2023-04-27 01:55:36,073 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/d5ade6ce7f5b42049d92ce2f3cbdb564 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d5ade6ce7f5b42049d92ce2f3cbdb564 2023-04-27 01:55:36,077 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d5ade6ce7f5b42049d92ce2f3cbdb564, entries=150, sequenceid=362, filesize=25.4 K 2023-04-27 01:55:36,078 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/41e79e35aab044eda59d0d2fe1a22020 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/41e79e35aab044eda59d0d2fe1a22020 2023-04-27 01:55:36,093 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/41e79e35aab044eda59d0d2fe1a22020, entries=150, sequenceid=362, filesize=11.9 K 2023-04-27 01:55:36,094 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/ec3ad296c9de43768556962b196b3447 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/ec3ad296c9de43768556962b196b3447 2023-04-27 01:55:36,105 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/ec3ad296c9de43768556962b196b3447, entries=150, sequenceid=362, filesize=11.9 K 2023-04-27 01:55:36,106 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b5f127509ba53373491a414d00be9fa1 in 933ms, sequenceid=362, compaction requested=true 2023-04-27 01:55:36,106 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:36,106 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:36,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:36,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:36,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:36,106 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:36,108 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,108 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,108 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:36,110 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:36,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:36,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:36,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b1a299[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:36,112 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:36,112 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,112 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:36,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:36,113 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:36,115 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,115 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:36,117 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:36,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:36,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:36,122 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:36,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:36,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:36,123 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:36,123 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:36,126 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,126 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:36,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,126 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:36,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:36,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:36,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:36,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:36,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:36,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:36,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:36,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:36,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:36,131 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:36,132 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:36,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:36,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:36,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:36,132 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:36,132 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:36,232 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:36,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:36,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:36,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:36,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:36,237 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:36,237 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:36,237 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:36,237 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:36,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,238 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:36,239 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:36,239 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:36,239 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,240 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:36,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:36,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:36,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:36,241 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:36,241 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:36,241 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:36,248 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:36,248 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:36,248 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:36,248 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:36,248 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:55:36,248 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:36,248 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:36,248 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:36,249 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:36,249 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:36,249 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:36,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:36,499 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:36,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560596527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560596528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560596529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560596530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560596531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560596632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560596638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560596638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40256 deadline: 1682560596638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560596642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,683 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427759d71707210494b9ec79110eef000d1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427759d71707210494b9ec79110eef000d1 2023-04-27 01:55:36,692 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/db77747241e347aea5221cebf68c0608 2023-04-27 01:55:36,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560596841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560596848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560596849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40256 deadline: 1682560596849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:36,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:36,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560596853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,118 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/f5c517c669e94596b5a9524edac447e2 2023-04-27 01:55:37,136 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/51088573226a4fe887c94ea462cf2110 2023-04-27 01:55:37,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/db77747241e347aea5221cebf68c0608 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/db77747241e347aea5221cebf68c0608 2023-04-27 01:55:37,143 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/db77747241e347aea5221cebf68c0608, entries=150, sequenceid=377, filesize=25.4 K 2023-04-27 01:55:37,144 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/f5c517c669e94596b5a9524edac447e2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/f5c517c669e94596b5a9524edac447e2 2023-04-27 01:55:37,147 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/f5c517c669e94596b5a9524edac447e2, entries=150, sequenceid=377, filesize=11.9 K 2023-04-27 01:55:37,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/51088573226a4fe887c94ea462cf2110 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/51088573226a4fe887c94ea462cf2110 2023-04-27 01:55:37,155 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/51088573226a4fe887c94ea462cf2110, entries=150, sequenceid=377, filesize=11.9 K 2023-04-27 01:55:37,156 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b5f127509ba53373491a414d00be9fa1 in 908ms, sequenceid=377, compaction requested=true 2023-04-27 01:55:37,156 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:37,156 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:37,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:37,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:37,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:37,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,157 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,158 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,158 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:37,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d2b3295[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:37,160 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:37,160 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,161 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:37,161 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:37,161 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:37,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:37,162 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,162 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,162 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,162 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:37,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:55:37,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:37,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:37,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:37,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:37,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:37,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:37,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:37,166 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:37,166 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:37,166 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:37,166 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:37,166 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:37,169 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,169 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,169 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,180 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,180 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:37,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:37,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,182 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:37,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,183 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:37,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:37,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560597179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560597181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40256 deadline: 1682560597182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560597183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560597186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,283 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:37,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:37,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:37,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:37,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:37,286 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:37,286 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,286 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:37,286 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:37,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,288 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,288 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,288 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:37,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:37,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:37,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:37,290 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,290 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:37,290 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:37,290 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:37,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:37,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:37,290 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:37,291 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,291 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,292 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,292 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c5f44b5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:37,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:37,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560597287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,295 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:37,296 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:37,296 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:37,296 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:37,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:37,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,297 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:37,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:37,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:37,300 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:37,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:37,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:37,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:37,300 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:37,301 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,301 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,301 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560597294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40256 deadline: 1682560597294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560597294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560597294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:37,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:37,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,311 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:37,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,311 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:37,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,411 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:37,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:37,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:37,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:37,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:37,416 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:37,416 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,416 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:37,416 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:37,416 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,417 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,419 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,419 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,419 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,419 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:37,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:37,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:37,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:37,420 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:37,420 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,420 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:37,420 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:37,420 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:37,420 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,421 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:37,421 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:37,421 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:37,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,423 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,423 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,423 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:37,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544a0573[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:37,425 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:37,425 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,426 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:37,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,426 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:37,428 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:37,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:37,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:37,431 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:37,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:37,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:37,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:37,431 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:37,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:37,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:37,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,437 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:37,437 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:37,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:37,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560597496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40256 deadline: 1682560597505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560597506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560597506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560597506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,537 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:37,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:37,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:37,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:37,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:37,540 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:37,540 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,540 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:37,540 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:37,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,542 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,542 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,542 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,542 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:37,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:37,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:37,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:37,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:37,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:37,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:37,546 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,547 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,547 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d56d009[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:37,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:37,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,550 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:37,551 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,551 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:37,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:37,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:37,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:37,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,552 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,552 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:37,554 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:37,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:37,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:37,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:37,555 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:37,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,566 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:37,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:37,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:37,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,566 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:37,566 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:37,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,597 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304270d1e0a399e704462af074ba8ed27238b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270d1e0a399e704462af074ba8ed27238b 2023-04-27 01:55:37,608 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=396, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/a02b36368395418eb7df85d1bf54ad4f 2023-04-27 01:55:37,666 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:37,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:37,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:37,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:37,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:37,671 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:37,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:37,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:37,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,672 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,672 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:37,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:37,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:37,674 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,674 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:37,674 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:37,674 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:37,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,676 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,677 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,677 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:37,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ddf8ac[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:37,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:37,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,679 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:37,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:37,681 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:37,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:37,685 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:37,685 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:37,685 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:37,685 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:37,685 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:37,686 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,686 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,687 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,687 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:37,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:37,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,690 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:37,691 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:37,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:37,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,791 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:37,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:37,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:37,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:37,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:37,794 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:37,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:37,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:37,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,795 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,800 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:37,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:37,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:37,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:37,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:37,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:37,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:37,802 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,802 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:37,802 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:37,802 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:37,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:37,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:37,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:37,804 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:37,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8165ce[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:37,806 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:37,806 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,807 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:37,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:37,808 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:37,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:37,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:37,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40294 deadline: 1682560597804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:37,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:37,811 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:37,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:37,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:37,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:37,811 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:37,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:37,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:37,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,819 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:37,819 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:37,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40336 deadline: 1682560597812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40256 deadline: 1682560597812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40306 deadline: 1682560597812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:37,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40264 deadline: 1682560597813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,919 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:37,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:37,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:37,922 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:37,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:37,922 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:37,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,924 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:37,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:37,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:37,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:37,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:37,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:37,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:37,928 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b6496d0[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:37,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:37,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:37,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:37,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:37,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:37,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,936 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:37,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:37,939 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:37,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:37,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:37,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:37,939 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:37,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,943 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:37,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:37,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:37,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:37,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:37,949 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:37,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:37,949 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:37,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:37,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:37,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:37,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:37,952 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:37,952 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:38,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/2da2466ef2e4406ea21fd8ab896b3d15 2023-04-27 01:55:38,052 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:38,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:38,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:38,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:38,065 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:38,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:38,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:38,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:38,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:38,066 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:38,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:38,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:38,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:38,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:38,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:38,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:38,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:38,078 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:38,078 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. started... 2023-04-27 01:55:38,078 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. as already flushing 2023-04-27 01:55:38,078 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:38,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:38,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:38,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:38,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:38,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:38,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:38,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:38,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:38,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2069c72d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:38,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:38,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,085 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:38,085 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:38,085 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:38,085 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,085 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:38,086 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,086 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:38,086 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:38,086 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:38,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:38,089 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:38,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:38,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:38,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:38,089 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:38,091 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,091 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:38,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,091 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:38,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/05e72f514f244edcbe6c039a50f148d1 2023-04-27 01:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:38,094 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:38,096 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/a02b36368395418eb7df85d1bf54ad4f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a02b36368395418eb7df85d1bf54ad4f 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:38,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:38,100 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:38,101 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:38,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:38,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:38,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:38,101 DEBUG [Thread-2180] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:38,101 DEBUG [Thread-2180] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:38,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a02b36368395418eb7df85d1bf54ad4f, entries=200, sequenceid=396, filesize=32.3 K 2023-04-27 01:55:38,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/2da2466ef2e4406ea21fd8ab896b3d15 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/2da2466ef2e4406ea21fd8ab896b3d15 2023-04-27 01:55:38,123 DEBUG [Thread-2187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43736f95 to 127.0.0.1:52731 2023-04-27 01:55:38,123 DEBUG [Thread-2187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/2da2466ef2e4406ea21fd8ab896b3d15, entries=150, sequenceid=396, filesize=11.9 K 2023-04-27 01:55:38,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/05e72f514f244edcbe6c039a50f148d1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/05e72f514f244edcbe6c039a50f148d1 2023-04-27 01:55:38,125 DEBUG [Thread-2189] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72eaaf6e to 127.0.0.1:52731 2023-04-27 01:55:38,125 DEBUG [Thread-2189] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,130 DEBUG [Thread-2181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4335c1f2 to 127.0.0.1:52731 2023-04-27 01:55:38,130 DEBUG [Thread-2181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/05e72f514f244edcbe6c039a50f148d1, entries=150, sequenceid=396, filesize=11.9 K 2023-04-27 01:55:38,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b5f127509ba53373491a414d00be9fa1 in 967ms, sequenceid=396, compaction requested=true 2023-04-27 01:55:38,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:38,131 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:38,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:38,132 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:55:38,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:38,132 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:38,138 DEBUG [Thread-2185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x678b4c3d to 127.0.0.1:52731 2023-04-27 01:55:38,138 DEBUG [Thread-2185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,138 DEBUG [Thread-2183] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bbc9fc8 to 127.0.0.1:52731 2023-04-27 01:55:38,138 DEBUG [Thread-2183] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,139 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:55:38,139 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/A is initiating minor compaction (all files) 2023-04-27 01:55:38,139 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/A in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:38,139 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f855732b98e04dafacc89e370fad95e1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9fc3185c715e4c428947f0ab225b6883, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d5ade6ce7f5b42049d92ce2f3cbdb564, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/db77747241e347aea5221cebf68c0608, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a02b36368395418eb7df85d1bf54ad4f] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=134.8 K 2023-04-27 01:55:38,140 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:38,140 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/B is initiating minor compaction (all files) 2023-04-27 01:55:38,140 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/B in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:38,140 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/a5a7fffd8c664ff8a86aae1aeee0a921, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9e529de4342a4370bb28b26e29753278, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/1635d28a996f4e5194e6c27e498baf8b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/41e79e35aab044eda59d0d2fe1a22020, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/f5c517c669e94596b5a9524edac447e2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/2da2466ef2e4406ea21fd8ab896b3d15] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=72.1 K 2023-04-27 01:55:38,141 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting a5a7fffd8c664ff8a86aae1aeee0a921, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682560533060 2023-04-27 01:55:38,142 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f855732b98e04dafacc89e370fad95e1, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682560533089 2023-04-27 01:55:38,142 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9e529de4342a4370bb28b26e29753278, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682560533089 2023-04-27 01:55:38,142 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9fc3185c715e4c428947f0ab225b6883, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682560533175 2023-04-27 01:55:38,142 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1635d28a996f4e5194e6c27e498baf8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682560533175 2023-04-27 01:55:38,142 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d5ade6ce7f5b42049d92ce2f3cbdb564, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560534355 2023-04-27 01:55:38,142 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 41e79e35aab044eda59d0d2fe1a22020, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560534355 2023-04-27 01:55:38,143 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting db77747241e347aea5221cebf68c0608, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682560535354 2023-04-27 01:55:38,143 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f5c517c669e94596b5a9524edac447e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682560535354 2023-04-27 01:55:38,143 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a02b36368395418eb7df85d1bf54ad4f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682560536519 2023-04-27 01:55:38,144 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2da2466ef2e4406ea21fd8ab896b3d15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682560536519 2023-04-27 01:55:38,153 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#B#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:38,156 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#A#compaction#369 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:38,201 DEBUG [Thread-2180] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:38,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:38,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:38,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:55:38,317 DEBUG [Thread-2178] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30951255 to 127.0.0.1:52731 2023-04-27 01:55:38,317 DEBUG [Thread-2178] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:38,323 DEBUG [Thread-2170] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a9450ee to 127.0.0.1:52731 2023-04-27 01:55:38,323 DEBUG [Thread-2170] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,323 DEBUG [Thread-2174] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d800352 to 127.0.0.1:52731 2023-04-27 01:55:38,323 DEBUG [Thread-2174] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,325 DEBUG [Thread-2176] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2329054b to 127.0.0.1:52731 2023-04-27 01:55:38,325 DEBUG [Thread-2176] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,326 DEBUG [Thread-2172] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x494529f9 to 127.0.0.1:52731 2023-04-27 01:55:38,326 DEBUG [Thread-2172] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 56 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1616 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 4848 rows 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1624 2023-04-27 01:55:38,326 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 4871 rows 2023-04-27 01:55:38,327 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1616 2023-04-27 01:55:38,327 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 4848 rows 2023-04-27 01:55:38,327 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1608 2023-04-27 01:55:38,327 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 4824 rows 2023-04-27 01:55:38,327 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1619 2023-04-27 01:55:38,327 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 4849 rows 2023-04-27 01:55:38,327 INFO [Listener at localhost/42495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 01:55:38,327 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x668bbba6 to 127.0.0.1:52731 2023-04-27 01:55:38,327 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:55:38,328 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304271be52022846e47008cc63101f670574c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304271be52022846e47008cc63101f670574c 2023-04-27 01:55:38,328 INFO [Listener at localhost/42495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 01:55:38,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 01:55:38,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:38,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 01:55:38,338 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560538338"}]},"ts":"1682560538338"} 2023-04-27 01:55:38,339 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 01:55:38,431 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 01:55:38,432 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, UNASSIGN}] 2023-04-27 01:55:38,434 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, UNASSIGN 2023-04-27 01:55:38,435 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:38,435 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560538435"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560538435"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560538435"}]},"ts":"1682560538435"} 2023-04-27 01:55:38,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:55:38,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 01:55:38,561 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/805ac096d95745218ccb26275c64136c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/805ac096d95745218ccb26275c64136c 2023-04-27 01:55:38,567 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b5f127509ba53373491a414d00be9fa1/B of b5f127509ba53373491a414d00be9fa1 into 805ac096d95745218ccb26275c64136c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:38,567 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:38,567 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/B, priority=10, startTime=1682560538132; duration=0sec 2023-04-27 01:55:38,567 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:38,567 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 01:55:38,569 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 01:55:38,569 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): b5f127509ba53373491a414d00be9fa1/C is initiating minor compaction (all files) 2023-04-27 01:55:38,569 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b5f127509ba53373491a414d00be9fa1/C in TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:38,569 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/08da19a445ec4978a1e828d7a7d3ce8a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/932c29acf459464a9d6e47b98f9ff396, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c5e5fc58618b4f37912400299e69edcb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/ec3ad296c9de43768556962b196b3447, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/51088573226a4fe887c94ea462cf2110, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/05e72f514f244edcbe6c039a50f148d1] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp, totalSize=72.1 K 2023-04-27 01:55:38,570 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 08da19a445ec4978a1e828d7a7d3ce8a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682560533060 2023-04-27 01:55:38,570 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 932c29acf459464a9d6e47b98f9ff396, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682560533089 2023-04-27 01:55:38,570 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c5e5fc58618b4f37912400299e69edcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682560533175 2023-04-27 01:55:38,571 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ec3ad296c9de43768556962b196b3447, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682560534355 2023-04-27 01:55:38,571 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 51088573226a4fe887c94ea462cf2110, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1682560535354 2023-04-27 01:55:38,571 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 05e72f514f244edcbe6c039a50f148d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1682560536519 2023-04-27 01:55:38,580 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): b5f127509ba53373491a414d00be9fa1#C#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-27 01:55:38,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:38,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b5f127509ba53373491a414d00be9fa1, disabling compactions & flushes 2023-04-27 01:55:38,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 2 compactions & cache flush to complete for region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 01:55:38,742 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/854a066bb9284f559127b28460270cb3 2023-04-27 01:55:38,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 01:55:38,973 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/f51834b3d7284fe7b8f98c2b7b550945 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f51834b3d7284fe7b8f98c2b7b550945 2023-04-27 01:55:38,976 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b5f127509ba53373491a414d00be9fa1/A of b5f127509ba53373491a414d00be9fa1 into f51834b3d7284fe7b8f98c2b7b550945(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:38,976 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:38,977 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/A, priority=11, startTime=1682560538131; duration=0sec 2023-04-27 01:55:38,977 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:38,988 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/bc58ecbff53c40469148a86858fc154b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/bc58ecbff53c40469148a86858fc154b 2023-04-27 01:55:38,991 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b5f127509ba53373491a414d00be9fa1/C of b5f127509ba53373491a414d00be9fa1 into bc58ecbff53c40469148a86858fc154b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:38,992 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:38,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:38,992 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1., storeName=b5f127509ba53373491a414d00be9fa1/C, priority=10, startTime=1682560538132; duration=0sec 2023-04-27 01:55:38,992 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:39,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/fd09da39092a46898706e6cb494ed624 2023-04-27 01:55:39,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 01:55:39,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/bc25d488396c48a89b5a9d620421058c 2023-04-27 01:55:39,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/854a066bb9284f559127b28460270cb3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/854a066bb9284f559127b28460270cb3 2023-04-27 01:55:39,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/854a066bb9284f559127b28460270cb3, entries=150, sequenceid=410, filesize=25.4 K 2023-04-27 01:55:39,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/fd09da39092a46898706e6cb494ed624 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/fd09da39092a46898706e6cb494ed624 2023-04-27 01:55:39,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/fd09da39092a46898706e6cb494ed624, entries=150, sequenceid=410, filesize=11.9 K 2023-04-27 01:55:39,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/bc25d488396c48a89b5a9d620421058c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/bc25d488396c48a89b5a9d620421058c 2023-04-27 01:55:39,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/bc25d488396c48a89b5a9d620421058c, entries=150, sequenceid=410, filesize=11.9 K 2023-04-27 01:55:39,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=26.84 KB/27480 for b5f127509ba53373491a414d00be9fa1 in 1261ms, sequenceid=410, compaction requested=false 2023-04-27 01:55:39,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:39,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. after waiting 0 ms 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:39,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b5f127509ba53373491a414d00be9fa1 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=A 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=B 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b5f127509ba53373491a414d00be9fa1, store=C 2023-04-27 01:55:39,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:39,683 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-27 01:55:39,696 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ec5e3506e6e24c418949284007b581af, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7d2fe5470ce44e398544c811faa51af3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/20decab11cfb4dc2a5f98ba664eee04e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/cc948ad734164dd3aec7b33c67a0bf2d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a30fb2f08c2c419f85efcfb6f91fa0cd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b8946eafdfdf48acb509eae9828d950a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d502d235b57342a6b51d5bcf54961b35, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/8cd75142a24b4ee4bce2e1119bdd9027, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7a0843ed25594282bfbe4b83c1921d19, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/c2592bbbc225433c9ab663291a8e83f3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b55ef3d3ca65419980f63f391646aaaa, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ce26187c80fb4450b0d333398bf19073, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/45786177c08848aaa2d1633e9aff386a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3ddf6d59437d468e9ceddc81080d9fc5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3cc4248549eb44fe86653c3e8b1acab9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/eaa2932d63794c86a53b0814cecf2d35, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f876eeecc7a1417093a0d8b7b1c94f72, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f55045c5438c4fe1a5b0a170edd22f40, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b669d1b87e6a44a4ad290ae457b7b804, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3e39559a7eea406aad6a7d2778b963dc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/fc1383b51b504c47bb3882f861c48198, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f855732b98e04dafacc89e370fad95e1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9fc3185c715e4c428947f0ab225b6883, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d5ade6ce7f5b42049d92ce2f3cbdb564, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/db77747241e347aea5221cebf68c0608, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a02b36368395418eb7df85d1bf54ad4f] to archive 2023-04-27 01:55:39,697 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:55:39,699 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ec5e3506e6e24c418949284007b581af to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ec5e3506e6e24c418949284007b581af 2023-04-27 01:55:39,700 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7d2fe5470ce44e398544c811faa51af3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7d2fe5470ce44e398544c811faa51af3 2023-04-27 01:55:39,701 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/20decab11cfb4dc2a5f98ba664eee04e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/20decab11cfb4dc2a5f98ba664eee04e 2023-04-27 01:55:39,702 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/cc948ad734164dd3aec7b33c67a0bf2d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/cc948ad734164dd3aec7b33c67a0bf2d 2023-04-27 01:55:39,703 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a30fb2f08c2c419f85efcfb6f91fa0cd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a30fb2f08c2c419f85efcfb6f91fa0cd 2023-04-27 01:55:39,703 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b8946eafdfdf48acb509eae9828d950a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b8946eafdfdf48acb509eae9828d950a 2023-04-27 01:55:39,704 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d502d235b57342a6b51d5bcf54961b35 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d502d235b57342a6b51d5bcf54961b35 2023-04-27 01:55:39,705 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/8cd75142a24b4ee4bce2e1119bdd9027 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/8cd75142a24b4ee4bce2e1119bdd9027 2023-04-27 01:55:39,706 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7a0843ed25594282bfbe4b83c1921d19 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/7a0843ed25594282bfbe4b83c1921d19 2023-04-27 01:55:39,706 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/c2592bbbc225433c9ab663291a8e83f3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/c2592bbbc225433c9ab663291a8e83f3 2023-04-27 01:55:39,707 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b55ef3d3ca65419980f63f391646aaaa to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b55ef3d3ca65419980f63f391646aaaa 2023-04-27 01:55:39,708 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ce26187c80fb4450b0d333398bf19073 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/ce26187c80fb4450b0d333398bf19073 2023-04-27 01:55:39,709 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/45786177c08848aaa2d1633e9aff386a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/45786177c08848aaa2d1633e9aff386a 2023-04-27 01:55:39,710 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3ddf6d59437d468e9ceddc81080d9fc5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3ddf6d59437d468e9ceddc81080d9fc5 2023-04-27 01:55:39,710 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3cc4248549eb44fe86653c3e8b1acab9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3cc4248549eb44fe86653c3e8b1acab9 2023-04-27 01:55:39,711 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/eaa2932d63794c86a53b0814cecf2d35 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/eaa2932d63794c86a53b0814cecf2d35 2023-04-27 01:55:39,712 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f876eeecc7a1417093a0d8b7b1c94f72 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f876eeecc7a1417093a0d8b7b1c94f72 2023-04-27 01:55:39,713 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f55045c5438c4fe1a5b0a170edd22f40 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f55045c5438c4fe1a5b0a170edd22f40 2023-04-27 01:55:39,714 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b669d1b87e6a44a4ad290ae457b7b804 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/b669d1b87e6a44a4ad290ae457b7b804 2023-04-27 01:55:39,715 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3e39559a7eea406aad6a7d2778b963dc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/3e39559a7eea406aad6a7d2778b963dc 2023-04-27 01:55:39,715 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/fc1383b51b504c47bb3882f861c48198 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/fc1383b51b504c47bb3882f861c48198 2023-04-27 01:55:39,716 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f855732b98e04dafacc89e370fad95e1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f855732b98e04dafacc89e370fad95e1 2023-04-27 01:55:39,717 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9fc3185c715e4c428947f0ab225b6883 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9fc3185c715e4c428947f0ab225b6883 2023-04-27 01:55:39,718 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d5ade6ce7f5b42049d92ce2f3cbdb564 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/d5ade6ce7f5b42049d92ce2f3cbdb564 2023-04-27 01:55:39,719 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/db77747241e347aea5221cebf68c0608 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/db77747241e347aea5221cebf68c0608 2023-04-27 01:55:39,720 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a02b36368395418eb7df85d1bf54ad4f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/a02b36368395418eb7df85d1bf54ad4f 2023-04-27 01:55:39,724 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c7e897701be5431493eca8bc8b1f722c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/dbaef951a7b24074ab4020b9a78fcc9f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/96e0f3aaabca4d18bde74c5a287dcb27, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/110d6b5209a246b2afdd0bbcb754757a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/afb242fc39f842ec87b51d1194fbb246, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c59ffb094e944994b9176a28391edee1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/48219db5b6a248e6875abab624b67838, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ab0be9be2784140b681bf03d4ed25f9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ef76e5d90b741eaacf273a47b9b5914, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/85e4656b645441dbbe74ea29904f5eb5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/11a2cbe177c548619b88719118b86e12, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/32f5705e72ee435297ee48749b8f1d5a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/22016fd6bf01444883623580ccd8ea5c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/cc0eba73b9834913aaf46a905762a2d0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/ccacc4bf5d9f4aeba493c5ec681d1019, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/186e055749624dfaa1998ae46d70b540, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/42318b871d144fa6a571d7539c286ed4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/531df00aa6794cd2927e9a998ce3e8fa, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/a5a7fffd8c664ff8a86aae1aeee0a921, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/b21eb3953a6b4bdea5508cd4089768d8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9e529de4342a4370bb28b26e29753278, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/1635d28a996f4e5194e6c27e498baf8b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/41e79e35aab044eda59d0d2fe1a22020, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/f5c517c669e94596b5a9524edac447e2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/2da2466ef2e4406ea21fd8ab896b3d15] to archive 2023-04-27 01:55:39,725 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:55:39,726 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c7e897701be5431493eca8bc8b1f722c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c7e897701be5431493eca8bc8b1f722c 2023-04-27 01:55:39,727 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/dbaef951a7b24074ab4020b9a78fcc9f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/dbaef951a7b24074ab4020b9a78fcc9f 2023-04-27 01:55:39,728 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/96e0f3aaabca4d18bde74c5a287dcb27 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/96e0f3aaabca4d18bde74c5a287dcb27 2023-04-27 01:55:39,729 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/110d6b5209a246b2afdd0bbcb754757a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/110d6b5209a246b2afdd0bbcb754757a 2023-04-27 01:55:39,729 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/afb242fc39f842ec87b51d1194fbb246 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/afb242fc39f842ec87b51d1194fbb246 2023-04-27 01:55:39,730 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c59ffb094e944994b9176a28391edee1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/c59ffb094e944994b9176a28391edee1 2023-04-27 01:55:39,731 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/48219db5b6a248e6875abab624b67838 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/48219db5b6a248e6875abab624b67838 2023-04-27 01:55:39,732 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ab0be9be2784140b681bf03d4ed25f9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ab0be9be2784140b681bf03d4ed25f9 2023-04-27 01:55:39,732 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ef76e5d90b741eaacf273a47b9b5914 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9ef76e5d90b741eaacf273a47b9b5914 2023-04-27 01:55:39,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/85e4656b645441dbbe74ea29904f5eb5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/85e4656b645441dbbe74ea29904f5eb5 2023-04-27 01:55:39,734 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/11a2cbe177c548619b88719118b86e12 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/11a2cbe177c548619b88719118b86e12 2023-04-27 01:55:39,735 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/32f5705e72ee435297ee48749b8f1d5a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/32f5705e72ee435297ee48749b8f1d5a 2023-04-27 01:55:39,735 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/0bbdd5fafdd14eb9ad32ec4dfb81d5d4 2023-04-27 01:55:39,736 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/22016fd6bf01444883623580ccd8ea5c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/22016fd6bf01444883623580ccd8ea5c 2023-04-27 01:55:39,737 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/cc0eba73b9834913aaf46a905762a2d0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/cc0eba73b9834913aaf46a905762a2d0 2023-04-27 01:55:39,738 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/ccacc4bf5d9f4aeba493c5ec681d1019 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/ccacc4bf5d9f4aeba493c5ec681d1019 2023-04-27 01:55:39,739 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/186e055749624dfaa1998ae46d70b540 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/186e055749624dfaa1998ae46d70b540 2023-04-27 01:55:39,739 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/42318b871d144fa6a571d7539c286ed4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/42318b871d144fa6a571d7539c286ed4 2023-04-27 01:55:39,740 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/531df00aa6794cd2927e9a998ce3e8fa to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/531df00aa6794cd2927e9a998ce3e8fa 2023-04-27 01:55:39,741 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/a5a7fffd8c664ff8a86aae1aeee0a921 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/a5a7fffd8c664ff8a86aae1aeee0a921 2023-04-27 01:55:39,742 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/b21eb3953a6b4bdea5508cd4089768d8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/b21eb3953a6b4bdea5508cd4089768d8 2023-04-27 01:55:39,742 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9e529de4342a4370bb28b26e29753278 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/9e529de4342a4370bb28b26e29753278 2023-04-27 01:55:39,743 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/1635d28a996f4e5194e6c27e498baf8b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/1635d28a996f4e5194e6c27e498baf8b 2023-04-27 01:55:39,744 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/41e79e35aab044eda59d0d2fe1a22020 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/41e79e35aab044eda59d0d2fe1a22020 2023-04-27 01:55:39,745 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/f5c517c669e94596b5a9524edac447e2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/f5c517c669e94596b5a9524edac447e2 2023-04-27 01:55:39,746 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/2da2466ef2e4406ea21fd8ab896b3d15 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/2da2466ef2e4406ea21fd8ab896b3d15 2023-04-27 01:55:39,750 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0c8223194e6a46578701919837eb66db, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9272833f438545ccaeeb334bd11a77ee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/deeadc77274844aaaf4ac693b9e2acbd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0d0a2190e3b34ee6aa302f6907bbb2d7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/698aa6533a7b4aa9b0fb0d56dfa8e16a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/b1a399f93ec947ad9d92fae5dd80c371, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/09bd3b4b91d04befaf89d9c7a897b920, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d881e1c0f1d74093a79249f726db49be, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/25de986d2ea545b5b9a9ff6c9e21a3ca, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/04fa7ee4836c4de288854fa655276e77, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9bcba7fea6934ac3b15a3c9973dd1374, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/45e27d1a56a349d88d3824283433cab1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d1e95477baf7457e8eb8c3d59d81b43b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97be6057afd44906b576f0ba1df5cb88, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/15445a5df5b64737a733ffb346f209b2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/1a012514a9fa4071b3ed16576fb62e9d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c7a5e0ebd0cd4b66818c924f82e4ab8f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97332050e5ec4e138d2ef5a8d5158bf0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/08da19a445ec4978a1e828d7a7d3ce8a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/36109ae633e1438395b7570d3b44f901, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/932c29acf459464a9d6e47b98f9ff396, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c5e5fc58618b4f37912400299e69edcb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/ec3ad296c9de43768556962b196b3447, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/51088573226a4fe887c94ea462cf2110, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/05e72f514f244edcbe6c039a50f148d1] to archive 2023-04-27 01:55:39,750 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:55:39,751 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0c8223194e6a46578701919837eb66db to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0c8223194e6a46578701919837eb66db 2023-04-27 01:55:39,752 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9272833f438545ccaeeb334bd11a77ee to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9272833f438545ccaeeb334bd11a77ee 2023-04-27 01:55:39,753 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/deeadc77274844aaaf4ac693b9e2acbd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/deeadc77274844aaaf4ac693b9e2acbd 2023-04-27 01:55:39,753 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0d0a2190e3b34ee6aa302f6907bbb2d7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/0d0a2190e3b34ee6aa302f6907bbb2d7 2023-04-27 01:55:39,754 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/698aa6533a7b4aa9b0fb0d56dfa8e16a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/698aa6533a7b4aa9b0fb0d56dfa8e16a 2023-04-27 01:55:39,755 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/b1a399f93ec947ad9d92fae5dd80c371 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/b1a399f93ec947ad9d92fae5dd80c371 2023-04-27 01:55:39,756 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/09bd3b4b91d04befaf89d9c7a897b920 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/09bd3b4b91d04befaf89d9c7a897b920 2023-04-27 01:55:39,756 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d881e1c0f1d74093a79249f726db49be to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d881e1c0f1d74093a79249f726db49be 2023-04-27 01:55:39,757 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/25de986d2ea545b5b9a9ff6c9e21a3ca to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/25de986d2ea545b5b9a9ff6c9e21a3ca 2023-04-27 01:55:39,758 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/04fa7ee4836c4de288854fa655276e77 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/04fa7ee4836c4de288854fa655276e77 2023-04-27 01:55:39,758 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9bcba7fea6934ac3b15a3c9973dd1374 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/9bcba7fea6934ac3b15a3c9973dd1374 2023-04-27 01:55:39,759 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/45e27d1a56a349d88d3824283433cab1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/45e27d1a56a349d88d3824283433cab1 2023-04-27 01:55:39,760 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d1e95477baf7457e8eb8c3d59d81b43b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/d1e95477baf7457e8eb8c3d59d81b43b 2023-04-27 01:55:39,760 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97be6057afd44906b576f0ba1df5cb88 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97be6057afd44906b576f0ba1df5cb88 2023-04-27 01:55:39,761 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/8ae8ca9a0c9449dbaa8aa23ff3143e1c 2023-04-27 01:55:39,762 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/15445a5df5b64737a733ffb346f209b2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/15445a5df5b64737a733ffb346f209b2 2023-04-27 01:55:39,762 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/1a012514a9fa4071b3ed16576fb62e9d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/1a012514a9fa4071b3ed16576fb62e9d 2023-04-27 01:55:39,763 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c7a5e0ebd0cd4b66818c924f82e4ab8f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c7a5e0ebd0cd4b66818c924f82e4ab8f 2023-04-27 01:55:39,764 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97332050e5ec4e138d2ef5a8d5158bf0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/97332050e5ec4e138d2ef5a8d5158bf0 2023-04-27 01:55:39,765 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/08da19a445ec4978a1e828d7a7d3ce8a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/08da19a445ec4978a1e828d7a7d3ce8a 2023-04-27 01:55:39,766 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/36109ae633e1438395b7570d3b44f901 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/36109ae633e1438395b7570d3b44f901 2023-04-27 01:55:39,766 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/932c29acf459464a9d6e47b98f9ff396 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/932c29acf459464a9d6e47b98f9ff396 2023-04-27 01:55:39,767 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c5e5fc58618b4f37912400299e69edcb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/c5e5fc58618b4f37912400299e69edcb 2023-04-27 01:55:39,768 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/ec3ad296c9de43768556962b196b3447 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/ec3ad296c9de43768556962b196b3447 2023-04-27 01:55:39,769 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/51088573226a4fe887c94ea462cf2110 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/51088573226a4fe887c94ea462cf2110 2023-04-27 01:55:39,770 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/05e72f514f244edcbe6c039a50f148d1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/05e72f514f244edcbe6c039a50f148d1 2023-04-27 01:55:39,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272a0105c205714280897424adf8f0f780 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272a0105c205714280897424adf8f0f780 2023-04-27 01:55:40,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=420, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/9ea89d530af84f97b31bd80bea71d360 2023-04-27 01:55:40,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 01:55:40,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/252b55cff3fb40cb84c65d1901ce21db 2023-04-27 01:55:41,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/5d8d0a12cd894f0389f0de5810791050 2023-04-27 01:55:41,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/A/9ea89d530af84f97b31bd80bea71d360 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9ea89d530af84f97b31bd80bea71d360 2023-04-27 01:55:41,225 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9ea89d530af84f97b31bd80bea71d360, entries=150, sequenceid=420, filesize=25.4 K 2023-04-27 01:55:41,225 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/B/252b55cff3fb40cb84c65d1901ce21db as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/252b55cff3fb40cb84c65d1901ce21db 2023-04-27 01:55:41,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/252b55cff3fb40cb84c65d1901ce21db, entries=150, sequenceid=420, filesize=11.9 K 2023-04-27 01:55:41,229 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/.tmp/C/5d8d0a12cd894f0389f0de5810791050 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/5d8d0a12cd894f0389f0de5810791050 2023-04-27 01:55:41,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/5d8d0a12cd894f0389f0de5810791050, entries=150, sequenceid=420, filesize=11.9 K 2023-04-27 01:55:41,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for b5f127509ba53373491a414d00be9fa1 in 1655ms, sequenceid=420, compaction requested=true 2023-04-27 01:55:41,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/recovered.edits/423.seqid, newMaxSeqId=423, maxSeqId=4 2023-04-27 01:55:41,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1. 2023-04-27 01:55:41,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b5f127509ba53373491a414d00be9fa1: 2023-04-27 01:55:41,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:41,242 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=b5f127509ba53373491a414d00be9fa1, regionState=CLOSED 2023-04-27 01:55:41,242 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560541242"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560541242"}]},"ts":"1682560541242"} 2023-04-27 01:55:41,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-27 01:55:41,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure b5f127509ba53373491a414d00be9fa1, server=jenkins-hbase4.apache.org,38085,1682560417985 in 2.8070 sec 2023-04-27 01:55:41,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-27 01:55:41,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b5f127509ba53373491a414d00be9fa1, UNASSIGN in 2.8130 sec 2023-04-27 01:55:41,247 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560541247"}]},"ts":"1682560541247"} 2023-04-27 01:55:41,248 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 01:55:41,249 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 01:55:41,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9210 sec 2023-04-27 01:55:42,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 01:55:42,443 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-27 01:55:42,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 01:55:42,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:42,446 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:42,447 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:42,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-27 01:55:42,453 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:42,454 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/recovered.edits] 2023-04-27 01:55:42,457 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/854a066bb9284f559127b28460270cb3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/854a066bb9284f559127b28460270cb3 2023-04-27 01:55:42,458 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9ea89d530af84f97b31bd80bea71d360 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/9ea89d530af84f97b31bd80bea71d360 2023-04-27 01:55:42,459 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f51834b3d7284fe7b8f98c2b7b550945 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/A/f51834b3d7284fe7b8f98c2b7b550945 2023-04-27 01:55:42,461 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/252b55cff3fb40cb84c65d1901ce21db to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/252b55cff3fb40cb84c65d1901ce21db 2023-04-27 01:55:42,462 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/805ac096d95745218ccb26275c64136c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/805ac096d95745218ccb26275c64136c 2023-04-27 01:55:42,463 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/fd09da39092a46898706e6cb494ed624 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/B/fd09da39092a46898706e6cb494ed624 2023-04-27 01:55:42,466 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/5d8d0a12cd894f0389f0de5810791050 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/5d8d0a12cd894f0389f0de5810791050 2023-04-27 01:55:42,467 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/bc25d488396c48a89b5a9d620421058c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/bc25d488396c48a89b5a9d620421058c 2023-04-27 01:55:42,468 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/bc58ecbff53c40469148a86858fc154b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/C/bc58ecbff53c40469148a86858fc154b 2023-04-27 01:55:42,471 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/recovered.edits/423.seqid to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1/recovered.edits/423.seqid 2023-04-27 01:55:42,471 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/b5f127509ba53373491a414d00be9fa1 2023-04-27 01:55:42,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:55:42,472 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 01:55:42,473 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-27 01:55:42,475 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270d1e0a399e704462af074ba8ed27238b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270d1e0a399e704462af074ba8ed27238b 2023-04-27 01:55:42,476 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304271be52022846e47008cc63101f670574c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304271be52022846e47008cc63101f670574c 2023-04-27 01:55:42,478 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272a0105c205714280897424adf8f0f780 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272a0105c205714280897424adf8f0f780 2023-04-27 01:55:42,479 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272ca915e71b314cb88753b286d6b99dec to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272ca915e71b314cb88753b286d6b99dec 2023-04-27 01:55:42,480 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042731c03a6d51fe47bf86e9e6c43e7aa405 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042731c03a6d51fe47bf86e9e6c43e7aa405 2023-04-27 01:55:42,481 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274fb2cf4bdb0e448895e4c6c0f14cc339 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274fb2cf4bdb0e448895e4c6c0f14cc339 2023-04-27 01:55:42,482 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042756aff53db9174299b9fdffc06cd7d3d0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042756aff53db9174299b9fdffc06cd7d3d0 2023-04-27 01:55:42,483 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276b6564a1e7a14095a96e0ed5799dacc0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276b6564a1e7a14095a96e0ed5799dacc0 2023-04-27 01:55:42,484 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427759d71707210494b9ec79110eef000d1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427759d71707210494b9ec79110eef000d1 2023-04-27 01:55:42,485 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042776a4ef6e46ec4004a3cf7f2890c15156 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042776a4ef6e46ec4004a3cf7f2890c15156 2023-04-27 01:55:42,486 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042780dc4b2c088544ceb6dd24bd465a9785 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042780dc4b2c088544ceb6dd24bd465a9785 2023-04-27 01:55:42,488 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042781c95994a3d6459bbfd2ae2ab6435172 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042781c95994a3d6459bbfd2ae2ab6435172 2023-04-27 01:55:42,489 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042784e880608fa54542b53e4e6f43323397 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042784e880608fa54542b53e4e6f43323397 2023-04-27 01:55:42,490 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042792ca08829fb440ee97a35c95142f2f2d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042792ca08829fb440ee97a35c95142f2f2d 2023-04-27 01:55:42,491 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042796feec413f0c403080edb270dc3f120a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042796feec413f0c403080edb270dc3f120a 2023-04-27 01:55:42,492 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042798f5ae42687f46938ab2850c693da46d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042798f5ae42687f46938ab2850c693da46d 2023-04-27 01:55:42,493 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279cbeacc833f3476a86067980c9d1d092 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279cbeacc833f3476a86067980c9d1d092 2023-04-27 01:55:42,494 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b68f7117924342c7b19620567505f273 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b68f7117924342c7b19620567505f273 2023-04-27 01:55:42,495 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c149d400bffb457fb7217ace76143fb6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c149d400bffb457fb7217ace76143fb6 2023-04-27 01:55:42,496 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c84ed25ef8fb4716b18440f345af3c97 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c84ed25ef8fb4716b18440f345af3c97 2023-04-27 01:55:42,498 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427ea5a0fa91d0c4d5fbc999cac0ed63b2e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427ea5a0fa91d0c4d5fbc999cac0ed63b2e 2023-04-27 01:55:42,499 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eb5d8a51ec5a453e90da146aee40aa6d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eb5d8a51ec5a453e90da146aee40aa6d 2023-04-27 01:55:42,500 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fe9ae382cef846199b7d72e66ed8e86c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fe9ae382cef846199b7d72e66ed8e86c 2023-04-27 01:55:42,501 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 01:55:42,504 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:42,505 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 01:55:42,507 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 01:55:42,508 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:42,508 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 01:55:42,508 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682560542508"}]},"ts":"9223372036854775807"} 2023-04-27 01:55:42,510 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 01:55:42,510 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b5f127509ba53373491a414d00be9fa1, NAME => 'TestAcidGuarantees,,1682560515036.b5f127509ba53373491a414d00be9fa1.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 01:55:42,510 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 01:55:42,510 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682560542510"}]},"ts":"9223372036854775807"} 2023-04-27 01:55:42,511 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 01:55:42,514 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:42,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 70 msec 2023-04-27 01:55:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-27 01:55:42,549 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-27 01:55:42,564 INFO [Listener at localhost/42495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=238 (was 227) - Thread LEAK? -, OpenFileDescriptor=560 (was 557) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=386 (was 324) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12713 (was 12006) - AvailableMemoryMB LEAK? - 2023-04-27 01:55:42,577 INFO [Listener at localhost/42495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=238, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=170, AvailableMemoryMB=12712 2023-04-27 01:55:42,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:55:42,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:55:42,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 01:55:42,582 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 01:55:42,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-27 01:55:42,582 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 01:55:42,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 01:55:42,583 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:42,584 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648 empty. 2023-04-27 01:55:42,584 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:42,584 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:55:42,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 01:55:42,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 01:55:42,995 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:55:42,997 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bf86160e96ad9ed4e3ec62cf4c749648, NAME => 'TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp 2023-04-27 01:55:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 01:55:43,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:55:43,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bf86160e96ad9ed4e3ec62cf4c749648, disabling compactions & flushes 2023-04-27 01:55:43,413 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. after waiting 0 ms 2023-04-27 01:55:43,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,413 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:43,415 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 01:55:43,416 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560543416"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560543416"}]},"ts":"1682560543416"} 2023-04-27 01:55:43,417 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 01:55:43,418 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 01:55:43,418 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560543418"}]},"ts":"1682560543418"} 2023-04-27 01:55:43,419 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 01:55:43,422 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf86160e96ad9ed4e3ec62cf4c749648, ASSIGN}] 2023-04-27 01:55:43,423 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf86160e96ad9ed4e3ec62cf4c749648, ASSIGN 2023-04-27 01:55:43,424 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf86160e96ad9ed4e3ec62cf4c749648, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=false 2023-04-27 01:55:43,575 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=bf86160e96ad9ed4e3ec62cf4c749648, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:43,576 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560543575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560543575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560543575"}]},"ts":"1682560543575"} 2023-04-27 01:55:43,577 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:55:43,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 01:55:43,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bf86160e96ad9ed4e3ec62cf4c749648, NAME => 'TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:55:43,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:55:43,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,733 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,735 DEBUG [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A 2023-04-27 01:55:43,735 DEBUG [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A 2023-04-27 01:55:43,735 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:43,735 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf86160e96ad9ed4e3ec62cf4c749648 columnFamilyName A 2023-04-27 01:55:43,736 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.HStore(310): Store=bf86160e96ad9ed4e3ec62cf4c749648/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:43,736 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,737 DEBUG [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B 2023-04-27 01:55:43,737 DEBUG [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B 2023-04-27 01:55:43,737 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:43,738 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf86160e96ad9ed4e3ec62cf4c749648 columnFamilyName B 2023-04-27 01:55:43,738 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.HStore(310): Store=bf86160e96ad9ed4e3ec62cf4c749648/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:43,738 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,739 DEBUG [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C 2023-04-27 01:55:43,740 DEBUG [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C 2023-04-27 01:55:43,740 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:55:43,740 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf86160e96ad9ed4e3ec62cf4c749648 columnFamilyName C 2023-04-27 01:55:43,741 INFO [StoreOpener-bf86160e96ad9ed4e3ec62cf4c749648-1] regionserver.HStore(310): Store=bf86160e96ad9ed4e3ec62cf4c749648/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:55:43,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:55:43,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:43,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:55:43,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bf86160e96ad9ed4e3ec62cf4c749648; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66005846, jitterRate=-0.016436249017715454}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:55:43,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:43,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., pid=49, masterSystemTime=1682560543729 2023-04-27 01:55:43,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,750 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:43,751 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=bf86160e96ad9ed4e3ec62cf4c749648, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:43,751 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560543751"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560543751"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560543751"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560543751"}]},"ts":"1682560543751"} 2023-04-27 01:55:43,753 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-27 01:55:43,753 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 in 175 msec 2023-04-27 01:55:43,754 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-27 01:55:43,755 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf86160e96ad9ed4e3ec62cf4c749648, ASSIGN in 332 msec 2023-04-27 01:55:43,755 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 01:55:43,755 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560543755"}]},"ts":"1682560543755"} 2023-04-27 01:55:43,756 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 01:55:43,758 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 01:55:43,759 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1790 sec 2023-04-27 01:55:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 01:55:44,688 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-27 01:55:44,689 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x6e0b6187 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,695 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5cd71f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,697 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,699 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,700 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 01:55:44,703 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 01:55:44,704 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x2ab56e54 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,708 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0735dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,709 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x15840296 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,713 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14c209bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,714 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x3816bdf5 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,723 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@db388d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,724 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x50150525 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,729 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5de1e3c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,730 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x028a3a77 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,735 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4e3b2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,736 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x1f65627a to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,743 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14f9adf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,743 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x15f74c78 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,751 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21359a0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,751 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x1401cac1 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,764 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bbac0b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,765 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x1f6e865f to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,771 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a300b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,771 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x48c22118 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:55:44,779 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23751a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:55:44,796 DEBUG [hconnection-0x1d52d96a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,796 DEBUG [hconnection-0x67bca903-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,806 DEBUG [hconnection-0x10a3143f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,804 DEBUG [hconnection-0x2eb51255-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,804 DEBUG [hconnection-0x2779f05a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,804 DEBUG [hconnection-0x284f45da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,807 DEBUG [hconnection-0x697a7736-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,807 DEBUG [hconnection-0x2a37cdd9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,807 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,808 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,813 DEBUG [hconnection-0x5916a954-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,815 DEBUG [hconnection-0x739a5bf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:55:44,815 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,820 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,820 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,822 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46396, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,823 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,824 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46414, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,824 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:44,829 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:55:44,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:44,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:44,837 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:44,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:44,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:44,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:44,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,837 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:44,842 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:44,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:44,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:44,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:44,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:44,848 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:44,848 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:44,848 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:44,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:44,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:55:44,857 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:44,857 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:44,857 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:44,857 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:44,857 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:44,858 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:44,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:44,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:44,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:44,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:44,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:44,863 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,863 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:44,863 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,863 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c74314[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:44,865 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:44,865 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,866 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:44,866 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,866 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:44,868 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:44,868 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:44,868 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:44,868 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:44,874 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:44,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:44,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:44,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:44,874 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:44,876 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,876 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,876 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:44,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:44,877 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:44,879 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560604877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560604878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560604880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:44,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560604881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:44,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560604881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:44,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:44,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:44,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:44,885 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:44,885 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:44,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:44,885 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:44,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:44,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:44,985 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:44,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:44,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560604981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560604981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560604981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560604982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:44,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560604982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:44,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:44,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:44,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:44,999 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:44,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:44,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:44,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:45,000 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,000 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,001 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,001 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,001 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,001 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:45,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:45,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,016 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:45,016 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:45,018 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621ceb17[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:45,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:45,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:45,032 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,033 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:45,033 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:45,033 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:45,033 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:45,033 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:45,033 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:45,033 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:45,034 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,034 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,034 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,035 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,036 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,038 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,038 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,038 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,039 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,038 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:45,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:45,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:45,039 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,039 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,039 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,039 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,039 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,039 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:45,039 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,139 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:45,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:45,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:45,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:45,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:45,151 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:45,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:45,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:45,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,159 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:45,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:45,168 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,168 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:45,168 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:45,168 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,170 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:45,170 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:45,171 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,171 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d113392[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:45,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:45,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:45,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:45,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:45,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:45,179 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:45,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:45,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:45,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:45,179 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:45,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,182 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,187 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:45,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:45,187 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:45,187 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:45,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:45,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560605188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560605189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560605190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560605190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560605190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,287 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:45,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:45,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:45,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:45,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:45,292 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:45,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:45,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:45,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,293 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,294 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,294 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:45,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:45,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:45,301 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:45,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:45,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/7314b61879f84f3fb8585415310d26d8 2023-04-27 01:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@532fcbb2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:45,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:45,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,312 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:45,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:45,314 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:45,320 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:45,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:45,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:45,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:45,320 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:45,322 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,322 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,322 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,322 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:45,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,326 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,327 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:45,327 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:45,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,427 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:45,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:45,431 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:45,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:45,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:45,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:45,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:45,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:45,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:45,448 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:45,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:45,448 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:45,449 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,451 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5cfd20[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:45,457 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:45,457 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,458 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:45,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:45,459 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:45,460 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:45,460 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:45,460 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:45,460 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:45,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:45,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:45,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:45,461 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:45,463 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,463 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,464 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,465 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,468 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,472 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:45,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:45,472 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,473 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:45,473 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:45,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560605495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560605497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560605497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560605497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:45,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560605498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,573 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:45,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:45,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:45,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:45,577 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:45,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:45,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:45,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,578 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,579 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:45,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:45,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:45,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:45,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:45,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:45,581 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,581 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:45,581 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:45,581 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:45,581 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:45,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:45,582 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:45,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,586 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,587 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,587 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1df58f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:45,588 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:45,588 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,590 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:45,590 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,590 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:45,591 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:45,591 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,591 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:45,591 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:45,595 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:45,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:45,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:45,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:45,595 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:45,598 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,598 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,598 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,599 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,600 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:45,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:45,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,603 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:45,603 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:45,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,604 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,603 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:45,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,704 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:45,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:45,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:45,707 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:45,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:45,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:45,708 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,708 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,709 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:45,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:45,710 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:45,710 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,710 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:45,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:45,711 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,711 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:45,711 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:45,711 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:45,712 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:45,713 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,713 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,714 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:45,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e24c4fd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:45,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:45,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,718 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:45,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:45,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:45,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,719 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:45,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:45,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,720 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:45,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:45,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:45,726 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:45,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:45,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:45,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:45,726 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:45,727 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,727 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,727 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:45,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,741 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/ca5dcc1a6a2d42c4ad360d551d4bf31f 2023-04-27 01:55:45,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,747 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,747 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:45,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:45,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,749 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:45,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:45,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:45,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:45,749 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,749 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:45,749 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:45,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/847858b7cbde4fa583e551c49018c87b 2023-04-27 01:55:45,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/7314b61879f84f3fb8585415310d26d8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/7314b61879f84f3fb8585415310d26d8 2023-04-27 01:55:45,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/7314b61879f84f3fb8585415310d26d8, entries=200, sequenceid=17, filesize=13.9 K 2023-04-27 01:55:45,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/ca5dcc1a6a2d42c4ad360d551d4bf31f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ca5dcc1a6a2d42c4ad360d551d4bf31f 2023-04-27 01:55:45,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ca5dcc1a6a2d42c4ad360d551d4bf31f, entries=150, sequenceid=17, filesize=11.6 K 2023-04-27 01:55:45,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/847858b7cbde4fa583e551c49018c87b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/847858b7cbde4fa583e551c49018c87b 2023-04-27 01:55:45,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/847858b7cbde4fa583e551c49018c87b, entries=150, sequenceid=17, filesize=11.6 K 2023-04-27 01:55:45,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=100.63 KB/103050 for bf86160e96ad9ed4e3ec62cf4c749648 in 936ms, sequenceid=17, compaction requested=false 2023-04-27 01:55:45,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:45,849 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:45,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:45,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:45,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:45,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:45,855 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:45,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:45,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:45,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,855 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,856 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:45,856 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:45,857 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:45,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:45,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:45,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:45,859 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:45,859 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:45,859 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:55:45,859 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:45,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:45,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:45,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:45,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:45,860 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:46,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:46,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:46,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560606013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560606015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560606016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560606016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560606017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560606118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560606121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560606123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560606123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560606123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,278 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/f5ab03601ed548d1b0a33e8abf6c84b2 2023-04-27 01:55:46,290 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/bf7f89686359479086592b02f5a47df3 2023-04-27 01:55:46,314 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/deb0a18c44ab4298bb1f95fb51fdc946 2023-04-27 01:55:46,319 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/f5ab03601ed548d1b0a33e8abf6c84b2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f5ab03601ed548d1b0a33e8abf6c84b2 2023-04-27 01:55:46,326 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f5ab03601ed548d1b0a33e8abf6c84b2, entries=150, sequenceid=35, filesize=11.6 K 2023-04-27 01:55:46,327 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/bf7f89686359479086592b02f5a47df3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/bf7f89686359479086592b02f5a47df3 2023-04-27 01:55:46,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560606323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560606325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560606326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560606326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560606326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,331 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/bf7f89686359479086592b02f5a47df3, entries=150, sequenceid=35, filesize=11.6 K 2023-04-27 01:55:46,332 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/deb0a18c44ab4298bb1f95fb51fdc946 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/deb0a18c44ab4298bb1f95fb51fdc946 2023-04-27 01:55:46,336 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/deb0a18c44ab4298bb1f95fb51fdc946, entries=150, sequenceid=35, filesize=11.6 K 2023-04-27 01:55:46,336 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for bf86160e96ad9ed4e3ec62cf4c749648 in 477ms, sequenceid=35, compaction requested=false 2023-04-27 01:55:46,337 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:46,337 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:46,339 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:46,340 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:46,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@637be4d0[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:46,342 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:46,342 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,343 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,343 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:46,343 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:46,343 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:46,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,343 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:46,344 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:46,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:46,348 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:46,348 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:46,348 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:46,348 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:46,349 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:46,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:46,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,350 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:46,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:46,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:46,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:46,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,353 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,354 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:46,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:46,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:46,358 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:46,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,359 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:46,359 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:46,359 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,359 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,359 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:46,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:46,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:46,359 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,359 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:46,359 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:46,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:46,459 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:46,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:46,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:46,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:46,462 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:46,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:46,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:46,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:46,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:46,464 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:46,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:46,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,465 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:46,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:46,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:46,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:46,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:46,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:46,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:46,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:46,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:46,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:46,467 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:46,468 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:46,468 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:55:46,468 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:46,468 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:46,468 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:46,468 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:46,468 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:46,468 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:46,631 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:46,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560606653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560606654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560606660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560606661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560606662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560606762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560606762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,766 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 01:55:46,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560606769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560606769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560606772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,882 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/01450f4240db45cb973d24ee8e68120f 2023-04-27 01:55:46,897 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/f081a4fc570245fe9fc3b5e3e24bff4b 2023-04-27 01:55:46,908 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/107b502d3a644a3d897fdf4d0b8c2cf7 2023-04-27 01:55:46,911 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/01450f4240db45cb973d24ee8e68120f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/01450f4240db45cb973d24ee8e68120f 2023-04-27 01:55:46,915 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/01450f4240db45cb973d24ee8e68120f, entries=150, sequenceid=49, filesize=11.6 K 2023-04-27 01:55:46,915 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/f081a4fc570245fe9fc3b5e3e24bff4b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f081a4fc570245fe9fc3b5e3e24bff4b 2023-04-27 01:55:46,918 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f081a4fc570245fe9fc3b5e3e24bff4b, entries=150, sequenceid=49, filesize=11.6 K 2023-04-27 01:55:46,919 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/107b502d3a644a3d897fdf4d0b8c2cf7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/107b502d3a644a3d897fdf4d0b8c2cf7 2023-04-27 01:55:46,922 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/107b502d3a644a3d897fdf4d0b8c2cf7, entries=150, sequenceid=49, filesize=11.6 K 2023-04-27 01:55:46,923 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for bf86160e96ad9ed4e3ec62cf4c749648 in 455ms, sequenceid=49, compaction requested=true 2023-04-27 01:55:46,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:46,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:46,924 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd6a062[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:46,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:46,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,927 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:46,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:46,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:46,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:46,935 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:46,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:46,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:46,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:46,935 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:46,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:46,937 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,938 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:46,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:46,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:46,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:46,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:46,942 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:46,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:46,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:46,941 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:46,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:46,942 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:46,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:46,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:46,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:46,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:46,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:55:46,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:46,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:46,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:46,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:46,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:46,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:46,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560606983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560606986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560606987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560606987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:46,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:46,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560606988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,043 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,046 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,048 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:47,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:47,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e72931a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:47,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:47,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,055 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:47,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,056 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:47,059 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:47,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:47,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:47,059 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:47,059 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:47,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,062 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:47,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:47,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,065 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:47,066 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:47,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:47,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560607090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560607095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560607095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560607095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560607096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,166 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,169 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,169 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,169 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,169 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,170 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,171 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,171 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,171 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,171 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:47,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:47,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619b105[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:47,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:47,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:47,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:47,180 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:47,182 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:47,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:47,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:47,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:47,183 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:47,184 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,184 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,184 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,185 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,186 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,188 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:47,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:47,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:47,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,189 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:47,189 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:47,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,189 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,289 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,295 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,296 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,296 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,297 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:47,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,298 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:47,298 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:47,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560607296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,301 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af88872[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:47,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:47,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,304 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:47,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:47,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,305 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,306 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560607300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560607301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560607302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560607302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:47,308 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:47,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:47,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:47,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:47,308 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:47,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,310 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,318 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:47,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:47,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,319 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:47,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,319 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,382 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/0b4d6149a5254de699b6ef1da56dd511 2023-04-27 01:55:47,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/feb65e14f6d348f28dced02a4c2c1f53 2023-04-27 01:55:47,419 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,422 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:47,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,427 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:47,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e3faff[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:47,429 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:47,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:47,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:47,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:47,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:47,436 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:47,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:47,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:47,436 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:47,436 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:47,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,451 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:47,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,452 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:47,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,452 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,552 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,555 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:47,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,560 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,560 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,561 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:47,561 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:47,563 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,563 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:47,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:47,566 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ef35d3a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:47,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:47,567 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,567 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:47,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:47,570 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:47,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:47,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:47,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:47,570 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:47,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,575 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,578 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:47,578 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:47,578 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:47,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560607601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560607608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560607608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560607610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:47,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560607611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,679 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,682 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:47,687 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:47,688 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a94478[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:47,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:47,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:47,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:47,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:47,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:47,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:47,697 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:47,698 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:47,698 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:47,698 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:47,698 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:47,699 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,699 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,699 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,701 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,706 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:47,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,706 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:47,706 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,706 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:47,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/d91e599372ac41238d97dee0e58e7cd9 2023-04-27 01:55:47,807 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,814 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/0b4d6149a5254de699b6ef1da56dd511 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0b4d6149a5254de699b6ef1da56dd511 2023-04-27 01:55:47,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,820 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:47,820 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:47,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:47,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:47,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:47,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0b4d6149a5254de699b6ef1da56dd511, entries=200, sequenceid=69, filesize=13.9 K 2023-04-27 01:55:47,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:47,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/feb65e14f6d348f28dced02a4c2c1f53 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/feb65e14f6d348f28dced02a4c2c1f53 2023-04-27 01:55:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6cefa7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:47,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:47,825 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/feb65e14f6d348f28dced02a4c2c1f53, entries=150, sequenceid=69, filesize=11.6 K 2023-04-27 01:55:47,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:47,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:47,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/d91e599372ac41238d97dee0e58e7cd9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/d91e599372ac41238d97dee0e58e7cd9 2023-04-27 01:55:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:47,829 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:47,833 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:47,833 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:47,833 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:47,833 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:47,833 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:47,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/d91e599372ac41238d97dee0e58e7cd9, entries=150, sequenceid=69, filesize=11.6 K 2023-04-27 01:55:47,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bf86160e96ad9ed4e3ec62cf4c749648 in 865ms, sequenceid=69, compaction requested=true 2023-04-27 01:55:47,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:47,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:47,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:47,834 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:55:47,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:47,834 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:55:47,836 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:55:47,836 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:55:47,836 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:55:47,836 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:55:47,836 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,836 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,836 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/7314b61879f84f3fb8585415310d26d8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f5ab03601ed548d1b0a33e8abf6c84b2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/01450f4240db45cb973d24ee8e68120f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0b4d6149a5254de699b6ef1da56dd511] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=51 K 2023-04-27 01:55:47,836 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ca5dcc1a6a2d42c4ad360d551d4bf31f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/bf7f89686359479086592b02f5a47df3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f081a4fc570245fe9fc3b5e3e24bff4b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/feb65e14f6d348f28dced02a4c2c1f53] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=46.4 K 2023-04-27 01:55:47,836 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 7314b61879f84f3fb8585415310d26d8, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682560544838 2023-04-27 01:55:47,836 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ca5dcc1a6a2d42c4ad360d551d4bf31f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682560544838 2023-04-27 01:55:47,837 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f5ab03601ed548d1b0a33e8abf6c84b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1682560544870 2023-04-27 01:55:47,837 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting bf7f89686359479086592b02f5a47df3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1682560544870 2023-04-27 01:55:47,837 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 01450f4240db45cb973d24ee8e68120f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682560546014 2023-04-27 01:55:47,837 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f081a4fc570245fe9fc3b5e3e24bff4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682560546014 2023-04-27 01:55:47,837 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 0b4d6149a5254de699b6ef1da56dd511, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682560546649 2023-04-27 01:55:47,837 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,837 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,837 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting feb65e14f6d348f28dced02a4c2c1f53, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682560546649 2023-04-27 01:55:47,837 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:47,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:47,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,845 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:47,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:47,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:47,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:47,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:47,849 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:47,849 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:47,852 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#391 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:47,852 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#390 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:47,860 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/07984863e3564e519ceb1357c7baa7fc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/07984863e3564e519ceb1357c7baa7fc 2023-04-27 01:55:47,864 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into 07984863e3564e519ceb1357c7baa7fc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:47,864 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:47,864 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=12, startTime=1682560547834; duration=0sec 2023-04-27 01:55:47,864 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:47,864 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:55:47,865 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:55:47,865 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:55:47,865 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,865 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/847858b7cbde4fa583e551c49018c87b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/deb0a18c44ab4298bb1f95fb51fdc946, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/107b502d3a644a3d897fdf4d0b8c2cf7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/d91e599372ac41238d97dee0e58e7cd9] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=46.4 K 2023-04-27 01:55:47,865 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 847858b7cbde4fa583e551c49018c87b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682560544838 2023-04-27 01:55:47,866 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting deb0a18c44ab4298bb1f95fb51fdc946, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1682560544870 2023-04-27 01:55:47,866 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 107b502d3a644a3d897fdf4d0b8c2cf7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682560546014 2023-04-27 01:55:47,866 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d91e599372ac41238d97dee0e58e7cd9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682560546649 2023-04-27 01:55:47,872 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:47,889 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/3fb7434e69624c96a89caf1ff3892fcd as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3fb7434e69624c96a89caf1ff3892fcd 2023-04-27 01:55:47,896 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 3fb7434e69624c96a89caf1ff3892fcd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:47,896 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:47,896 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=12, startTime=1682560547834; duration=0sec 2023-04-27 01:55:47,896 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:47,949 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:47,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:47,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:47,954 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:47,954 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:47,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:47,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,957 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:47,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:47,957 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:47,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:47,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:47,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:47,960 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:47,961 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:47,961 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:55:47,961 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:47,961 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:47,961 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:47,961 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:47,961 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:47,961 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:47,979 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/a846e5b16a24469699ca9fa7088102e5 2023-04-27 01:55:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:48,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:48,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560608130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560608133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560608134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560608135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560608138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560608239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560608239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560608241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560608242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560608242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,261 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/6de73f0e987e4fdb830346344e4f2a91 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6de73f0e987e4fdb830346344e4f2a91 2023-04-27 01:55:48,265 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into 6de73f0e987e4fdb830346344e4f2a91(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:48,265 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:48,265 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=12, startTime=1682560547834; duration=0sec 2023-04-27 01:55:48,265 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:48,391 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/77b3b54e763243ad99374ae64c45bacb 2023-04-27 01:55:48,403 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/eb3b273532294a8a87eafe2ecbab3dea 2023-04-27 01:55:48,414 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/a846e5b16a24469699ca9fa7088102e5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a846e5b16a24469699ca9fa7088102e5 2023-04-27 01:55:48,417 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a846e5b16a24469699ca9fa7088102e5, entries=100, sequenceid=83, filesize=9.3 K 2023-04-27 01:55:48,418 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/77b3b54e763243ad99374ae64c45bacb as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/77b3b54e763243ad99374ae64c45bacb 2023-04-27 01:55:48,422 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/77b3b54e763243ad99374ae64c45bacb, entries=100, sequenceid=83, filesize=9.3 K 2023-04-27 01:55:48,422 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/eb3b273532294a8a87eafe2ecbab3dea as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/eb3b273532294a8a87eafe2ecbab3dea 2023-04-27 01:55:48,425 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/eb3b273532294a8a87eafe2ecbab3dea, entries=100, sequenceid=83, filesize=9.3 K 2023-04-27 01:55:48,426 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bf86160e96ad9ed4e3ec62cf4c749648 in 465ms, sequenceid=83, compaction requested=false 2023-04-27 01:55:48,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:48,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:48,427 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5033de37[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:48,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:48,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:48,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:48,432 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:48,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:48,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:48,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,433 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:48,437 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:48,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:48,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:48,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:48,437 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:48,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,440 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,442 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:48,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,444 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:48,444 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:48,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,445 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,445 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,445 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,448 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-27 01:55:48,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:48,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:48,449 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:48,449 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:48,449 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:48,449 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:48,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:48,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560608453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560608457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560608458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560608458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560608461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,545 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:48,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:48,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:48,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:48,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:48,548 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:48,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:48,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:48,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,549 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,549 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,550 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,550 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:48,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:48,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:48,551 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,551 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:48,551 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:48,551 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:48,553 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,553 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@443d4386[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:48,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:48,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:48,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:48,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:48,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:48,558 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:48,561 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:48,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:48,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:48,561 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:48,561 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:48,562 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,562 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,562 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:48,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:48,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:48,566 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,567 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,567 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:48,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,567 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560608562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560608564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560608564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560608564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560608569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,667 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:48,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:48,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:48,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:48,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:48,671 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:48,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:48,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:48,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,676 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:48,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:48,680 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,680 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:48,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:48,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:48,682 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:48,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bb845fd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:48,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:48,685 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,685 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:48,686 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:48,689 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:48,689 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:48,689 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:48,689 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:48,689 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:48,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,695 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:48,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:48,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,696 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:48,696 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:48,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560608769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560608770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560608770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560608771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:48,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560608777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,796 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:48,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:48,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:48,800 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:48,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:48,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:48,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,800 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,801 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,802 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,802 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:48,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:48,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:48,803 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,803 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:48,803 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:48,803 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:48,805 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:48,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f58afa[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:48,807 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:48,810 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:48,813 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:48,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:48,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:48,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:48,814 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:48,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:48,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,819 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,819 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:48,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:48,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/f980016be92249ca84472dbfbe6b203e 2023-04-27 01:55:48,919 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:48,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:48,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:48,923 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:48,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:48,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:48,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,925 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,925 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,926 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:48,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:48,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:48,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:48,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:48,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:48,927 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:48,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:48,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f92c55[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:48,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:48,930 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,931 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,931 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:48,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:48,935 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:48,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:48,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:48,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:48,936 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:48,938 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,938 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,938 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:48,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:48,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,940 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:48,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:48,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:48,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:48,943 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:48,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:48,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:48,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:48,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:48,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:48,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:48,943 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:48,943 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:49,044 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:49,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:49,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:49,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:49,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:49,047 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:49,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:49,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:49,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,048 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,049 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,049 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:49,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:49,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:49,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:49,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:49,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:49,051 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:49,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7776659f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:49,062 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:49,062 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:49,063 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:49,064 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:49,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:49,065 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:49,067 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:49,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:49,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:49,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:49,068 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:49,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,073 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,077 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,077 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,077 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,077 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,077 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:49,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:49,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,077 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:49,077 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:49,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560609074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560609074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560609076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560609076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560609084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,178 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:49,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:49,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:49,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:49,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:49,181 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:49,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:49,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:49,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,183 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:49,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:49,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:49,185 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,185 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:49,185 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:49,185 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:49,188 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,189 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:49,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e685121[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:49,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:49,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:49,191 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,191 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:49,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:49,196 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:49,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:49,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:49,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:49,196 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:49,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,200 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:49,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:49,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,202 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,203 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:49,203 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:49,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/7f046340181b4b978cdf8c73aa716a67 2023-04-27 01:55:49,303 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:49,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:49,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:49,306 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:49,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:49,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:49,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,308 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:49,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:49,310 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,310 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:49,310 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:49,310 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,312 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4571b64c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:49,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:49,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,315 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:49,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:49,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:49,320 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:49,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:49,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:49,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:49,320 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:49,321 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,321 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,321 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,322 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:49,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,327 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,327 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:49,328 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:49,428 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:49,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:49,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:49,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:49,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:49,431 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:49,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:49,434 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,434 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:49,435 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:49,435 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:49,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:49,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55faa133[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:49,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:49,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,438 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:49,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:49,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:49,445 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:49,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:49,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:49,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:49,445 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:49,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,448 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,450 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:49,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:49,450 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,451 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:49,451 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:49,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:49,551 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:49,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:49,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:49,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:49,555 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:49,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:49,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:49,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,556 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:49,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:49,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:49,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:49,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:49,558 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:49,561 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11646c8e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:49,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:49,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:49,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:49,564 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:49,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:49,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:49,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:49,569 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:49,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:49,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:49,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:49,570 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:49,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,571 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,573 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:49,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:49,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,577 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,578 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:49,578 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:49,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:49,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560609581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560609581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560609582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560609583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:49,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560609588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/c4b87ab2d2d34ab2964dbaa620fe2ff1 2023-04-27 01:55:49,678 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:49,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:49,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/f980016be92249ca84472dbfbe6b203e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f980016be92249ca84472dbfbe6b203e 2023-04-27 01:55:49,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:49,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:49,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:49,682 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:49,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:49,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:49,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f980016be92249ca84472dbfbe6b203e, entries=200, sequenceid=105, filesize=13.9 K 2023-04-27 01:55:49,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/7f046340181b4b978cdf8c73aa716a67 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/7f046340181b4b978cdf8c73aa716a67 2023-04-27 01:55:49,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:49,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:49,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:49,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:49,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:49,686 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:49,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/7f046340181b4b978cdf8c73aa716a67, entries=150, sequenceid=105, filesize=11.6 K 2023-04-27 01:55:49,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/c4b87ab2d2d34ab2964dbaa620fe2ff1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/c4b87ab2d2d34ab2964dbaa620fe2ff1 2023-04-27 01:55:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d39c6da[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:49,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:49,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:49,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/c4b87ab2d2d34ab2964dbaa620fe2ff1, entries=150, sequenceid=105, filesize=11.6 K 2023-04-27 01:55:49,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:49,694 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for bf86160e96ad9ed4e3ec62cf4c749648 in 1246ms, sequenceid=105, compaction requested=true 2023-04-27 01:55:49,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:49,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:49,694 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:49,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:49,694 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:49,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:49,695 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:49,695 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:55:49,695 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33452 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:49,695 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,695 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:55:49,695 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/07984863e3564e519ceb1357c7baa7fc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a846e5b16a24469699ca9fa7088102e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f980016be92249ca84472dbfbe6b203e] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=35.0 K 2023-04-27 01:55:49,695 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,696 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6de73f0e987e4fdb830346344e4f2a91, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/77b3b54e763243ad99374ae64c45bacb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/7f046340181b4b978cdf8c73aa716a67] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=32.7 K 2023-04-27 01:55:49,696 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 07984863e3564e519ceb1357c7baa7fc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682560546649 2023-04-27 01:55:49,696 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6de73f0e987e4fdb830346344e4f2a91, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682560546649 2023-04-27 01:55:49,696 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a846e5b16a24469699ca9fa7088102e5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682560546985 2023-04-27 01:55:49,696 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 77b3b54e763243ad99374ae64c45bacb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682560546985 2023-04-27 01:55:49,696 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f980016be92249ca84472dbfbe6b203e, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682560548127 2023-04-27 01:55:49,697 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7f046340181b4b978cdf8c73aa716a67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682560548127 2023-04-27 01:55:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:49,697 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:49,702 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:49,702 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:49,702 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:49,702 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:49,702 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:49,703 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#399 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:49,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,704 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:49,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,705 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#400 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,711 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:49,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:49,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:49,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:49,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:49,712 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:49,712 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:49,730 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/d9dbf656c98e4958a9e876336c8f12cc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9dbf656c98e4958a9e876336c8f12cc 2023-04-27 01:55:49,735 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into d9dbf656c98e4958a9e876336c8f12cc(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:49,735 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:49,735 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=13, startTime=1682560549694; duration=0sec 2023-04-27 01:55:49,735 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:49,735 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:49,736 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33452 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:49,736 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:55:49,736 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,736 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3fb7434e69624c96a89caf1ff3892fcd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/eb3b273532294a8a87eafe2ecbab3dea, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/c4b87ab2d2d34ab2964dbaa620fe2ff1] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=32.7 K 2023-04-27 01:55:49,737 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3fb7434e69624c96a89caf1ff3892fcd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682560546649 2023-04-27 01:55:49,737 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting eb3b273532294a8a87eafe2ecbab3dea, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682560546985 2023-04-27 01:55:49,737 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c4b87ab2d2d34ab2964dbaa620fe2ff1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682560548127 2023-04-27 01:55:49,742 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:49,812 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:49,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:49,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:49,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:49,815 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:49,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:49,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:49,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,816 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:49,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:49,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:49,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:49,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:49,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:49,818 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:49,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:49,819 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:55:49,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:49,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:49,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:49,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:49,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:49,819 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:49,834 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/71a9bae941be4b4cb28f2b0b3d91f6c8 2023-04-27 01:55:49,850 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/59611f5ef72a4f009f813afb6be5c299 2023-04-27 01:55:50,122 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/73ff59072e1844fb944f106f54f0d640 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/73ff59072e1844fb944f106f54f0d640 2023-04-27 01:55:50,126 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into 73ff59072e1844fb944f106f54f0d640(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:50,126 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:50,126 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=13, startTime=1682560549694; duration=0sec 2023-04-27 01:55:50,126 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:50,158 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/956d0ad87c16490db6939b2ee05982b0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/956d0ad87c16490db6939b2ee05982b0 2023-04-27 01:55:50,162 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 956d0ad87c16490db6939b2ee05982b0(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:50,162 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:50,162 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=13, startTime=1682560549694; duration=0sec 2023-04-27 01:55:50,162 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:50,263 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/2ed7f3c0664b46348e716cb4178a086d 2023-04-27 01:55:50,267 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/71a9bae941be4b4cb28f2b0b3d91f6c8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/71a9bae941be4b4cb28f2b0b3d91f6c8 2023-04-27 01:55:50,271 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/71a9bae941be4b4cb28f2b0b3d91f6c8, entries=150, sequenceid=117, filesize=11.6 K 2023-04-27 01:55:50,271 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/59611f5ef72a4f009f813afb6be5c299 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/59611f5ef72a4f009f813afb6be5c299 2023-04-27 01:55:50,275 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/59611f5ef72a4f009f813afb6be5c299, entries=150, sequenceid=117, filesize=11.6 K 2023-04-27 01:55:50,275 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/2ed7f3c0664b46348e716cb4178a086d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2ed7f3c0664b46348e716cb4178a086d 2023-04-27 01:55:50,279 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2ed7f3c0664b46348e716cb4178a086d, entries=150, sequenceid=117, filesize=11.6 K 2023-04-27 01:55:50,280 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for bf86160e96ad9ed4e3ec62cf4c749648 in 461ms, sequenceid=117, compaction requested=false 2023-04-27 01:55:50,280 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:50,280 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:50,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6532d6e7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:50,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:50,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,286 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:50,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:50,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:50,291 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:50,291 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:50,291 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:50,291 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:50,291 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:50,297 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,297 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,297 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,299 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,301 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:50,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,302 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:50,302 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:50,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:50,402 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:50,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:50,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:50,406 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:50,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:50,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:50,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,407 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,407 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,408 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,408 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:50,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:50,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:50,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:50,409 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,409 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:50,409 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:50,409 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:50,411 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,411 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,412 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,412 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:50,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6541004b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:50,413 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:50,413 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:50,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:50,425 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:50,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:50,428 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:50,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:50,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:50,428 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:50,428 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:50,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:50,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:50,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:50,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,435 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:50,435 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:50,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,535 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:50,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:50,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:50,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:50,538 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:50,538 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,538 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:50,538 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:50,539 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,539 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,541 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,541 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:50,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:50,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:50,543 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:50,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:50,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:50,544 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:50,545 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,545 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,546 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:50,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f31071[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:50,547 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:50,547 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,548 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:50,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,549 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:50,550 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:50,553 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:50,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:50,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:50,553 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:50,553 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:50,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,555 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,555 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,556 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:50,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:50,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:50,560 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:50,560 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:50,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:50,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:50,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:50,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:50,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:50,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:50,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:50,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560550615, max timestamp=1682560550628Num uniques 859; 2023-04-27 01:55:50,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:50,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560550615, max timestamp=1682560550628Num uniques 859; 2023-04-27 01:55:50,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560550615, max timestamp=1682560550628Num uniques 859; 2023-04-27 01:55:50,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:50,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:50,660 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:50,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560610649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560610649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560610649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:50,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560610661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:50,664 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:50,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:50,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:50,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,664 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,665 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,665 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,666 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,666 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:50,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:50,667 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,667 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:50,667 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:50,667 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:50,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,669 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:50,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341f1343[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:50,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:50,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560610662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,672 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:50,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,673 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:50,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:50,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:50,676 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:50,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:50,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:50,676 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:50,676 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:50,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,678 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:50,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:50,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,690 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:50,690 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:50,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560610763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560610763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560610763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560610764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560610771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,790 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:50,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:50,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:50,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:50,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:50,794 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:50,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:50,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:50,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,795 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,795 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,796 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,796 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:50,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:50,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:50,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:50,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:50,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:50,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:50,797 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,797 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:50,798 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:50,798 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:50,799 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,800 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:50,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e97ec9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:50,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:50,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:50,802 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,803 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:50,803 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:50,805 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:50,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:50,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:50,809 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:50,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:50,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:50,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:50,809 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:50,811 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,811 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,811 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:50,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:50,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,816 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:50,816 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:50,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:50,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,916 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:50,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:50,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:50,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:50,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:50,919 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:50,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:50,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:50,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,920 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,920 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:50,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:50,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:50,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,924 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:50,924 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:50,924 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:50,926 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:50,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ac76d27[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:50,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:50,928 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,929 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:50,929 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,929 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,930 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:50,932 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:50,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:50,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:50,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:50,932 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:50,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,933 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:50,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:50,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:50,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:50,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:50,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:50,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:50,940 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:50,940 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:50,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560610969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560610969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560610969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560610970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:50,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:50,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560610978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/8247508290294de3b9af11817a8d45c7 2023-04-27 01:55:51,040 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:51,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:51,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:51,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:51,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:51,045 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:51,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:51,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:51,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,046 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,046 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,047 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:51,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:51,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:51,049 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:51,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:51,050 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:51,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,052 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:51,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65f88096[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:51,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:51,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,055 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:51,055 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:51,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:51,058 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:51,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:51,061 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:51,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:51,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:51,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:51,061 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:51,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,063 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,064 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:51,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:51,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:51,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,069 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:51,069 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:51,069 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:51,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,169 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:51,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:51,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:51,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:51,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:51,173 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:51,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:51,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:51,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,181 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,181 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,182 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:51,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:51,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:51,183 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,183 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:51,183 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:51,183 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:51,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:51,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b10261[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:51,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:51,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:51,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:51,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:51,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:51,196 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:51,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:51,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:51,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:51,196 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:51,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,201 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,203 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:51,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:51,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:51,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,203 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:51,203 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560611280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560611281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560611282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560611283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560611284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,303 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:51,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:51,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:51,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:51,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:51,307 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:51,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:51,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:51,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,308 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:51,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:51,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:51,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25a23ea8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:51,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:51,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:51,318 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,318 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:51,319 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:51,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,319 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:51,322 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:51,322 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:51,322 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:51,322 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:51,322 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:51,324 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,324 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,324 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,329 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:51,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:51,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:51,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,329 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:51,330 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:51,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,330 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/6efbe9b006dc4394b15244252f2f9864 2023-04-27 01:55:51,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/fdcd279820264838ac611f34a37f2f73 2023-04-27 01:55:51,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/8247508290294de3b9af11817a8d45c7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8247508290294de3b9af11817a8d45c7 2023-04-27 01:55:51,430 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:51,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:51,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8247508290294de3b9af11817a8d45c7, entries=150, sequenceid=132, filesize=11.8 K 2023-04-27 01:55:51,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:51,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:51,434 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:51,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:51,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:51,434 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,435 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/6efbe9b006dc4394b15244252f2f9864 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6efbe9b006dc4394b15244252f2f9864 2023-04-27 01:55:51,437 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:51,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:51,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6efbe9b006dc4394b15244252f2f9864, entries=150, sequenceid=132, filesize=11.8 K 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:51,439 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,439 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:51,439 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:51,439 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/fdcd279820264838ac611f34a37f2f73 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fdcd279820264838ac611f34a37f2f73 2023-04-27 01:55:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:51,440 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fdcd279820264838ac611f34a37f2f73, entries=150, sequenceid=132, filesize=11.8 K 2023-04-27 01:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for bf86160e96ad9ed4e3ec62cf4c749648 in 846ms, sequenceid=132, compaction requested=true 2023-04-27 01:55:51,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,444 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:51,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:51,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:51,444 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:51,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341dd16e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:51,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:51,445 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,445 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:51,445 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:51,445 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:55:51,445 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:55:51,445 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,445 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,445 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,446 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/73ff59072e1844fb944f106f54f0d640, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/71a9bae941be4b4cb28f2b0b3d91f6c8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8247508290294de3b9af11817a8d45c7] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=35.2 K 2023-04-27 01:55:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,445 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9dbf656c98e4958a9e876336c8f12cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/59611f5ef72a4f009f813afb6be5c299, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6efbe9b006dc4394b15244252f2f9864] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=35.2 K 2023-04-27 01:55:51,446 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,446 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:51,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:51,447 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 73ff59072e1844fb944f106f54f0d640, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682560548127 2023-04-27 01:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:51,447 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d9dbf656c98e4958a9e876336c8f12cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682560548127 2023-04-27 01:55:51,447 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 59611f5ef72a4f009f813afb6be5c299, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682560548456 2023-04-27 01:55:51,448 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 71a9bae941be4b4cb28f2b0b3d91f6c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682560548456 2023-04-27 01:55:51,448 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6efbe9b006dc4394b15244252f2f9864, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560550594 2023-04-27 01:55:51,450 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8247508290294de3b9af11817a8d45c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560550594 2023-04-27 01:55:51,459 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:51,461 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,461 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:51,461 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:51,461 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,461 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:51,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:51,466 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:51,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:51,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:51,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:51,466 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:51,468 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,468 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,469 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:51,471 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,473 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,475 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,475 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,475 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,475 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,475 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:51,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:51,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:51,475 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,475 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,476 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:51,476 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,476 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:51,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:51,476 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:51,490 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/e31796ad8e4c49e18f86598315ec6b80 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e31796ad8e4c49e18f86598315ec6b80 2023-04-27 01:55:51,493 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into e31796ad8e4c49e18f86598315ec6b80(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:51,493 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:51,493 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=13, startTime=1682560551444; duration=0sec 2023-04-27 01:55:51,493 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:51,494 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:51,494 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:51,495 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:55:51,495 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,495 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/956d0ad87c16490db6939b2ee05982b0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2ed7f3c0664b46348e716cb4178a086d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fdcd279820264838ac611f34a37f2f73] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=35.2 K 2023-04-27 01:55:51,495 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 956d0ad87c16490db6939b2ee05982b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682560548127 2023-04-27 01:55:51,495 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2ed7f3c0664b46348e716cb4178a086d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682560548456 2023-04-27 01:55:51,496 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting fdcd279820264838ac611f34a37f2f73, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560550594 2023-04-27 01:55:51,499 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#410 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:51,514 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/4b42284b23bc47bcb969ede77755edeb as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4b42284b23bc47bcb969ede77755edeb 2023-04-27 01:55:51,517 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 4b42284b23bc47bcb969ede77755edeb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:51,517 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:51,517 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=13, startTime=1682560551444; duration=0sec 2023-04-27 01:55:51,517 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:51,576 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:51,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:51,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:51,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:51,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:51,579 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:51,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:51,579 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:51,580 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,580 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:51,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:51,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:51,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:51,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:51,583 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:51,583 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:51,583 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-27 01:55:51,583 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:51,583 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:51,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:51,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:51,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:51,584 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:51,591 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/636a1bb211654aa4a31d3fc325656d7d 2023-04-27 01:55:51,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:51,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:51,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560611794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560611795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560611796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560611804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560611805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,882 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/56bcd2306e5345a5a76dbdd0b10feeb0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/56bcd2306e5345a5a76dbdd0b10feeb0 2023-04-27 01:55:51,886 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into 56bcd2306e5345a5a76dbdd0b10feeb0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:51,886 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:51,886 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=13, startTime=1682560551444; duration=0sec 2023-04-27 01:55:51,886 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:51,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560611906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560611906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560611906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560611906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:51,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560611911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,006 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/011f4dcf3d344d578886197f0115781e 2023-04-27 01:55:52,020 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/ff73f3a4c92d4f89825c3a41d63ea5a4 2023-04-27 01:55:52,025 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/636a1bb211654aa4a31d3fc325656d7d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/636a1bb211654aa4a31d3fc325656d7d 2023-04-27 01:55:52,029 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/636a1bb211654aa4a31d3fc325656d7d, entries=150, sequenceid=169, filesize=11.8 K 2023-04-27 01:55:52,029 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/011f4dcf3d344d578886197f0115781e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/011f4dcf3d344d578886197f0115781e 2023-04-27 01:55:52,032 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/011f4dcf3d344d578886197f0115781e, entries=150, sequenceid=169, filesize=11.8 K 2023-04-27 01:55:52,033 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/ff73f3a4c92d4f89825c3a41d63ea5a4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ff73f3a4c92d4f89825c3a41d63ea5a4 2023-04-27 01:55:52,036 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ff73f3a4c92d4f89825c3a41d63ea5a4, entries=150, sequenceid=169, filesize=11.8 K 2023-04-27 01:55:52,037 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for bf86160e96ad9ed4e3ec62cf4c749648 in 454ms, sequenceid=169, compaction requested=false 2023-04-27 01:55:52,037 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:52,037 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,037 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:52,037 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:52,037 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,037 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:52,037 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:52,041 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,042 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15400b4b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:52,043 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:52,043 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,044 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:52,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,045 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:52,046 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:52,050 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:52,050 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:52,050 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:52,050 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:52,050 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:52,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,052 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,051 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,054 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,055 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,058 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,058 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,058 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:52,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:52,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,058 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,059 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:52,059 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:52,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:52,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,059 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:52,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:55:52,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:52,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:52,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:52,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:52,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:52,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:52,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/c8a824b5979544b5941df5188cf2bc5b 2023-04-27 01:55:52,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/16ceeb0a426b4875b3d96c92c00c7b37 2023-04-27 01:55:52,139 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/1d73eaba34d541f490537ca6facd822a 2023-04-27 01:55:52,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560552123, max timestamp=1682560552141Num uniques 859; 2023-04-27 01:55:52,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:52,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560552123, max timestamp=1682560552141Num uniques 859; 2023-04-27 01:55:52,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560552123, max timestamp=1682560552141Num uniques 859; 2023-04-27 01:55:52,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:52,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:52,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/c8a824b5979544b5941df5188cf2bc5b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c8a824b5979544b5941df5188cf2bc5b 2023-04-27 01:55:52,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c8a824b5979544b5941df5188cf2bc5b, entries=150, sequenceid=181, filesize=11.8 K 2023-04-27 01:55:52,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/16ceeb0a426b4875b3d96c92c00c7b37 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/16ceeb0a426b4875b3d96c92c00c7b37 2023-04-27 01:55:52,151 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/16ceeb0a426b4875b3d96c92c00c7b37, entries=150, sequenceid=181, filesize=11.8 K 2023-04-27 01:55:52,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/1d73eaba34d541f490537ca6facd822a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1d73eaba34d541f490537ca6facd822a 2023-04-27 01:55:52,155 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1d73eaba34d541f490537ca6facd822a, entries=150, sequenceid=181, filesize=11.8 K 2023-04-27 01:55:52,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=87.22 KB/89310 for bf86160e96ad9ed4e3ec62cf4c749648 in 42ms, sequenceid=181, compaction requested=true 2023-04-27 01:55:52,155 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:52,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:52,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:52,155 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:52,155 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:52,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:52,157 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:52,157 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:55:52,157 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:52,157 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,157 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:55:52,157 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/56bcd2306e5345a5a76dbdd0b10feeb0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/636a1bb211654aa4a31d3fc325656d7d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c8a824b5979544b5941df5188cf2bc5b] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=35.6 K 2023-04-27 01:55:52,157 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,157 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e31796ad8e4c49e18f86598315ec6b80, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/011f4dcf3d344d578886197f0115781e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/16ceeb0a426b4875b3d96c92c00c7b37] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=35.6 K 2023-04-27 01:55:52,157 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 56bcd2306e5345a5a76dbdd0b10feeb0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560550594 2023-04-27 01:55:52,158 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e31796ad8e4c49e18f86598315ec6b80, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560550594 2023-04-27 01:55:52,158 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 636a1bb211654aa4a31d3fc325656d7d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682560550647 2023-04-27 01:55:52,158 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 011f4dcf3d344d578886197f0115781e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682560550647 2023-04-27 01:55:52,158 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c8a824b5979544b5941df5188cf2bc5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682560551802 2023-04-27 01:55:52,158 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 16ceeb0a426b4875b3d96c92c00c7b37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682560551802 2023-04-27 01:55:52,159 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:52,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:52,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=93.93 KB heapSize=251.39 KB 2023-04-27 01:55:52,161 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:52,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:52,161 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:52,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:52,161 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:52,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:52,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:52,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:52,168 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:52,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:52,168 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:52,171 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,171 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,171 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:52,171 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:52,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,173 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:52,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:52,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:52,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:52,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560612181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560612183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560612189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560612190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:52,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:52,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:52,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:52,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:52,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:52,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560612192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:52,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c3a188[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:52,207 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:52,207 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,208 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/975d3ed8c1d04aefba73cb472d937fba as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/975d3ed8c1d04aefba73cb472d937fba 2023-04-27 01:55:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:52,209 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,209 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:52,209 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/c2e548392c224e7293d461f6a55971d4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c2e548392c224e7293d461f6a55971d4 2023-04-27 01:55:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:52,211 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,211 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,212 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into 975d3ed8c1d04aefba73cb472d937fba(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:52,212 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:52,212 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=13, startTime=1682560552155; duration=0sec 2023-04-27 01:55:52,213 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:52,213 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,213 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into c2e548392c224e7293d461f6a55971d4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:52,213 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:52,213 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=13, startTime=1682560552155; duration=0sec 2023-04-27 01:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,213 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:52,214 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,214 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:55:52,214 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,214 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4b42284b23bc47bcb969ede77755edeb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ff73f3a4c92d4f89825c3a41d63ea5a4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1d73eaba34d541f490537ca6facd822a] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=35.6 K 2023-04-27 01:55:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,214 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 4b42284b23bc47bcb969ede77755edeb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1682560550594 2023-04-27 01:55:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:52,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:52,215 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:52,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:52,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:52,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:52,215 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ff73f3a4c92d4f89825c3a41d63ea5a4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682560550647 2023-04-27 01:55:52,215 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:52,215 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1d73eaba34d541f490537ca6facd822a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682560551802 2023-04-27 01:55:52,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,217 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,217 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,218 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,220 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:52,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:52,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,220 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,220 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,221 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,220 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,221 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,221 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:52,221 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:52,221 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,221 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,221 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,221 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:52,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,223 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:52,231 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/6691f53ed89043829fb0e82980ea806b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/6691f53ed89043829fb0e82980ea806b 2023-04-27 01:55:52,235 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 6691f53ed89043829fb0e82980ea806b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:52,235 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:52,235 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=13, startTime=1682560552155; duration=0sec 2023-04-27 01:55:52,235 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:52,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560612291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560612295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560612295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560612295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560612305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,321 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:52,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:52,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:52,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:52,324 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:52,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:52,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:52,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,325 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,333 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,333 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,334 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,334 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:52,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:52,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:52,335 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,335 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:52,335 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:52,335 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:52,335 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:52,338 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:52,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@208ec6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:52,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:52,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,341 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:52,341 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:52,341 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,341 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:52,343 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:52,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:52,347 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:52,347 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:52,347 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:52,347 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:52,347 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:52,348 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,348 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,349 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,354 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,354 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,355 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,355 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,355 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,355 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:52,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:52,355 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,355 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,355 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,355 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,355 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:52,355 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:52,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,455 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:52,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:52,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:52,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:52,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:52,461 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:52,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:52,461 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:52,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,462 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,463 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,463 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:52,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:52,465 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,465 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:52,465 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:52,465 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:52,466 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,467 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,467 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aeca90d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:52,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:52,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,470 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:52,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:52,474 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:52,477 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:52,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:52,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:52,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:52,477 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:52,483 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,483 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,483 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,485 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,487 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:52,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:52,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,487 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,487 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:52,488 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:52,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560612496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560612499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560612499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560612501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560612507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/8a67789e77a048d5b43ad11d2ed4d91c 2023-04-27 01:55:52,588 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:52,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:52,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/85fc2ba2ff1145d1b2736c028d8376df 2023-04-27 01:55:52,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:52,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:52,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:52,594 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:52,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:52,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:52,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,598 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,598 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,598 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,598 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:52,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:52,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:52,599 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:52,599 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,599 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:52,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:52,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:52,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:52,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:52,602 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,602 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,603 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:52,603 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/b53fb8bca0ab4e6982a49e952fe9382f 2023-04-27 01:55:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe3fb48[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:52,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:52,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,607 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:52,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,607 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:52,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/8a67789e77a048d5b43ad11d2ed4d91c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8a67789e77a048d5b43ad11d2ed4d91c 2023-04-27 01:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,608 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:52,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:52,611 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:52,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:52,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:52,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:52,611 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:52,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8a67789e77a048d5b43ad11d2ed4d91c, entries=250, sequenceid=209, filesize=16.4 K 2023-04-27 01:55:52,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:52,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:52,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/85fc2ba2ff1145d1b2736c028d8376df as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/85fc2ba2ff1145d1b2736c028d8376df 2023-04-27 01:55:52,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,615 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,616 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/85fc2ba2ff1145d1b2736c028d8376df, entries=150, sequenceid=209, filesize=11.8 K 2023-04-27 01:55:52,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/b53fb8bca0ab4e6982a49e952fe9382f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b53fb8bca0ab4e6982a49e952fe9382f 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:52,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,621 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:52,621 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,621 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:52,621 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:52,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:52,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b53fb8bca0ab4e6982a49e952fe9382f, entries=150, sequenceid=209, filesize=11.8 K 2023-04-27 01:55:52,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~251.34 KB/257376, currentSize=100.63 KB/103050 for bf86160e96ad9ed4e3ec62cf4c749648 in 463ms, sequenceid=209, compaction requested=false 2023-04-27 01:55:52,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:52,721 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:52,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:52,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:52,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:52,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:52,724 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:52,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:52,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:52,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,726 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:52,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:52,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:52,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:52,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:52,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:52,728 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:52,728 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:52,728 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:55:52,728 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:52,728 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:52,744 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/097dfc3f75f54840ae734118f1be1cf0 2023-04-27 01:55:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:52,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:52,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560612816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560612823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560612823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560612824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560612824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560612926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560612931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560612931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:52,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560612934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560613131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560613137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560613138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560613138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,164 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/e3d585e12c11413abf2c2804f89ceb47 2023-04-27 01:55:53,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560613328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560613436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560613439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560613441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560613442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,579 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/af5268f550654350bbafde45b35094fe 2023-04-27 01:55:53,583 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/097dfc3f75f54840ae734118f1be1cf0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/097dfc3f75f54840ae734118f1be1cf0 2023-04-27 01:55:53,586 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/097dfc3f75f54840ae734118f1be1cf0, entries=150, sequenceid=230, filesize=11.8 K 2023-04-27 01:55:53,587 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/e3d585e12c11413abf2c2804f89ceb47 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e3d585e12c11413abf2c2804f89ceb47 2023-04-27 01:55:53,590 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e3d585e12c11413abf2c2804f89ceb47, entries=150, sequenceid=230, filesize=11.8 K 2023-04-27 01:55:53,591 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/af5268f550654350bbafde45b35094fe as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/af5268f550654350bbafde45b35094fe 2023-04-27 01:55:53,594 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/af5268f550654350bbafde45b35094fe, entries=150, sequenceid=230, filesize=11.8 K 2023-04-27 01:55:53,595 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for bf86160e96ad9ed4e3ec62cf4c749648 in 867ms, sequenceid=230, compaction requested=true 2023-04-27 01:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:53,596 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6605ccdd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:53,598 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:53,598 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,599 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:53,600 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,600 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:53,601 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:53,601 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:53,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:53,603 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:53,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:53,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:53,603 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:53,603 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:53,607 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,607 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:53,607 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:53,608 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:53,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:53,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:53,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:53,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,611 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,611 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:53,611 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:53,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:53,712 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:53,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:53,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:53,715 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:53,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:53,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:53,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:53,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,715 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,716 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:53,716 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,717 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:53,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:53,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:53,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:53,718 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:53,718 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:53,718 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:55:53,718 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:53,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:53,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:53,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:53,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:53,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:53,729 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/2e7860a417854486a4a80bfe0bb67d57 2023-04-27 01:55:53,744 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/c3e2345825624f4b9258b38a10d4f8fb 2023-04-27 01:55:53,754 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/f67932f1e6e64e4ab006d3ca4eacb52b 2023-04-27 01:55:53,760 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/2e7860a417854486a4a80bfe0bb67d57 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e7860a417854486a4a80bfe0bb67d57 2023-04-27 01:55:53,764 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e7860a417854486a4a80bfe0bb67d57, entries=150, sequenceid=244, filesize=11.8 K 2023-04-27 01:55:53,765 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/c3e2345825624f4b9258b38a10d4f8fb as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/c3e2345825624f4b9258b38a10d4f8fb 2023-04-27 01:55:53,768 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/c3e2345825624f4b9258b38a10d4f8fb, entries=150, sequenceid=244, filesize=11.8 K 2023-04-27 01:55:53,769 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/f67932f1e6e64e4ab006d3ca4eacb52b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f67932f1e6e64e4ab006d3ca4eacb52b 2023-04-27 01:55:53,774 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f67932f1e6e64e4ab006d3ca4eacb52b, entries=150, sequenceid=244, filesize=11.8 K 2023-04-27 01:55:53,774 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for bf86160e96ad9ed4e3ec62cf4c749648 in 56ms, sequenceid=244, compaction requested=true 2023-04-27 01:55:53,774 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:53,774 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:53,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d73865[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:53,778 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:53,778 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,780 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:53,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,781 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:53,782 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:53,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:53,784 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:53,784 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:53,784 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:53,784 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:53,784 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:53,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:53,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,789 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,792 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,792 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:53,792 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:53,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:53,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:53,792 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:53,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:53,792 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,793 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:53,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,793 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:53,793 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:53,893 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:53,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:53,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:53,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:53,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:53,896 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:53,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:53,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:53,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:53,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:53,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:53,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:53,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:53,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:53,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:53,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:53,900 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:53,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:53,902 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,902 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcc51f3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:53,904 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:53,904 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,905 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,905 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,905 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:53,906 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:53,906 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:53,907 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,907 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,907 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:53,907 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:53,910 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:53,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:53,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:53,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:53,910 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:53,911 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:53,912 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,912 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:53,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:53,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:53,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,914 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,916 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:53,916 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:53,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:53,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:53,916 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:53,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:53,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:53,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:53,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:53,917 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:53,917 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:53,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:53,957 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:53,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:53,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:53,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:53,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:53,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:53,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:53,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:53,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560613989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560613991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560553987, max timestamp=1682560553996Num uniques 859; 2023-04-27 01:55:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560553987, max timestamp=1682560553996Num uniques 859; 2023-04-27 01:55:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560553987, max timestamp=1682560553996Num uniques 859; 2023-04-27 01:55:53,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:55:54,017 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,020 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,020 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,022 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,024 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,024 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,024 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:54,024 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:54,025 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:54,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d25c8b9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:54,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:54,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,028 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:54,029 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,029 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:54,031 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:54,034 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:54,034 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:54,034 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:54,034 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:54,034 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:54,035 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,035 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,035 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,036 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,038 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,040 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:54,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,040 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:54,040 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:54,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560614058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560614061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560614099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560614099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,141 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,143 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,145 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,145 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:54,150 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,150 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@458e8ee4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:54,153 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:54,153 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,154 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:54,154 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,154 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:54,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:54,157 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:54,157 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:54,158 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:54,158 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:54,158 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:54,159 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,159 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,159 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,160 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,166 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:54,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,167 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:54,167 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,167 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:54,167 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560614168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560614168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,267 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,271 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,273 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,273 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,273 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,273 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,274 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,275 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,275 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,275 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:54,275 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:54,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:54,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e419220[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:54,279 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:54,279 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:54,283 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,283 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:54,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:54,283 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:54,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:54,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:54,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:54,287 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:54,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:54,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:54,287 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:54,287 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:54,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:54,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,292 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,293 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,293 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,293 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:54,293 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:54,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46424 deadline: 1682560614305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46420 deadline: 1682560614306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560614336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/4c78374d4d8644018e6c7a48ca70d581 2023-04-27 01:55:54,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/2c1a66729dff4217bb9ba1a5f2b16f11 2023-04-27 01:55:54,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46404 deadline: 1682560614377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46414 deadline: 1682560614377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/a8365797acff482abe1fde143a2f4ddc 2023-04-27 01:55:54,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/4c78374d4d8644018e6c7a48ca70d581 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4c78374d4d8644018e6c7a48ca70d581 2023-04-27 01:55:54,393 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4c78374d4d8644018e6c7a48ca70d581, entries=250, sequenceid=256, filesize=16.4 K 2023-04-27 01:55:54,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,396 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,396 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/2c1a66729dff4217bb9ba1a5f2b16f11 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2c1a66729dff4217bb9ba1a5f2b16f11 2023-04-27 01:55:54,396 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,396 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,397 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,397 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,400 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2c1a66729dff4217bb9ba1a5f2b16f11, entries=150, sequenceid=256, filesize=11.8 K 2023-04-27 01:55:54,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:54,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/a8365797acff482abe1fde143a2f4ddc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/a8365797acff482abe1fde143a2f4ddc 2023-04-27 01:55:54,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,404 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21bd1bd2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:54,407 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:54,407 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/a8365797acff482abe1fde143a2f4ddc, entries=150, sequenceid=256, filesize=11.8 K 2023-04-27 01:55:54,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:54,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for bf86160e96ad9ed4e3ec62cf4c749648 in 452ms, sequenceid=256, compaction requested=true 2023-04-27 01:55:54,409 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:54,409 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:54,409 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:54,409 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:55:54,409 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:55:54,409 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:54,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:54,411 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:54,411 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60624 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:55:54,411 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 70184 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:55:54,411 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:55:54,411 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:55:54,411 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,411 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,411 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/975d3ed8c1d04aefba73cb472d937fba, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/85fc2ba2ff1145d1b2736c028d8376df, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e3d585e12c11413abf2c2804f89ceb47, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/c3e2345825624f4b9258b38a10d4f8fb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2c1a66729dff4217bb9ba1a5f2b16f11] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=59.2 K 2023-04-27 01:55:54,411 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c2e548392c224e7293d461f6a55971d4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8a67789e77a048d5b43ad11d2ed4d91c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/097dfc3f75f54840ae734118f1be1cf0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e7860a417854486a4a80bfe0bb67d57, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4c78374d4d8644018e6c7a48ca70d581] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=68.5 K 2023-04-27 01:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,412 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 975d3ed8c1d04aefba73cb472d937fba, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682560551802 2023-04-27 01:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,412 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c2e548392c224e7293d461f6a55971d4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682560551802 2023-04-27 01:55:54,412 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 85fc2ba2ff1145d1b2736c028d8376df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682560552149 2023-04-27 01:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,413 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8a67789e77a048d5b43ad11d2ed4d91c, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682560552141 2023-04-27 01:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,413 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 097dfc3f75f54840ae734118f1be1cf0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682560552169 2023-04-27 01:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,413 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e3d585e12c11413abf2c2804f89ceb47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682560552169 2023-04-27 01:55:54,413 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2e7860a417854486a4a80bfe0bb67d57, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682560552814 2023-04-27 01:55:54,413 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c3e2345825624f4b9258b38a10d4f8fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682560552814 2023-04-27 01:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,414 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 4c78374d4d8644018e6c7a48ca70d581, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682560553948 2023-04-27 01:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,414 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2c1a66729dff4217bb9ba1a5f2b16f11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682560553948 2023-04-27 01:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:54,414 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:54,415 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:54,415 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:54,415 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:54,415 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:54,421 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,421 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,421 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,424 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,428 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#432 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:54,428 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#433 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:54,429 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:54,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:54,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,430 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:54,430 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:54,438 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/2538ad199cbb4e1aa2f0b336be278b7f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2538ad199cbb4e1aa2f0b336be278b7f 2023-04-27 01:55:54,441 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into 2538ad199cbb4e1aa2f0b336be278b7f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:54,441 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:54,442 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=11, startTime=1682560554409; duration=0sec 2023-04-27 01:55:54,442 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:54,442 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:55:54,443 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60624 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:55:54,443 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:55:54,443 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,443 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/6691f53ed89043829fb0e82980ea806b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b53fb8bca0ab4e6982a49e952fe9382f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/af5268f550654350bbafde45b35094fe, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f67932f1e6e64e4ab006d3ca4eacb52b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/a8365797acff482abe1fde143a2f4ddc] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=59.2 K 2023-04-27 01:55:54,443 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6691f53ed89043829fb0e82980ea806b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682560551802 2023-04-27 01:55:54,444 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b53fb8bca0ab4e6982a49e952fe9382f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682560552149 2023-04-27 01:55:54,449 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting af5268f550654350bbafde45b35094fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682560552169 2023-04-27 01:55:54,452 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f67932f1e6e64e4ab006d3ca4eacb52b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682560552814 2023-04-27 01:55:54,454 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting a8365797acff482abe1fde143a2f4ddc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682560553948 2023-04-27 01:55:54,463 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#434 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:54,476 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/84090761666b4393a8bab35774ca2fc2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/84090761666b4393a8bab35774ca2fc2 2023-04-27 01:55:54,481 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 84090761666b4393a8bab35774ca2fc2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:54,481 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:54,481 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=11, startTime=1682560554409; duration=0sec 2023-04-27 01:55:54,481 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:54,531 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,534 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,534 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,534 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,534 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,535 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,535 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,536 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,536 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,536 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,538 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,538 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,538 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-27 01:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:54,539 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:54,555 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/a557a1ffee3c4bba8b35136c13a09f48 2023-04-27 01:55:54,565 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/b6b24a59ef05413093fab6ee3ed276d5 2023-04-27 01:55:54,581 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/b5064c7dbab546bab75b76fd8277b78a 2023-04-27 01:55:54,585 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/a557a1ffee3c4bba8b35136c13a09f48 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a557a1ffee3c4bba8b35136c13a09f48 2023-04-27 01:55:54,591 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a557a1ffee3c4bba8b35136c13a09f48, entries=150, sequenceid=293, filesize=11.9 K 2023-04-27 01:55:54,592 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/b6b24a59ef05413093fab6ee3ed276d5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b6b24a59ef05413093fab6ee3ed276d5 2023-04-27 01:55:54,596 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b6b24a59ef05413093fab6ee3ed276d5, entries=150, sequenceid=293, filesize=11.9 K 2023-04-27 01:55:54,600 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/b5064c7dbab546bab75b76fd8277b78a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b5064c7dbab546bab75b76fd8277b78a 2023-04-27 01:55:54,604 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b5064c7dbab546bab75b76fd8277b78a, entries=150, sequenceid=293, filesize=11.9 K 2023-04-27 01:55:54,605 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=0 B/0 for bf86160e96ad9ed4e3ec62cf4c749648 in 67ms, sequenceid=293, compaction requested=false 2023-04-27 01:55:54,605 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:54,605 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,608 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe6f5c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:54,612 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:54,612 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,613 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:54,614 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,614 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:54,615 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:54,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:54,622 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:54,622 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:54,622 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:54,622 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:54,622 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:54,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,623 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,625 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:54,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:54,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:54,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:54,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:54,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:54,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:54,630 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,630 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,631 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:54,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:54,631 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:54,631 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:54,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:54,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560554657, max timestamp=1682560554696Num uniques 859; 2023-04-27 01:55:54,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560554657, max timestamp=1682560554696Num uniques 859; 2023-04-27 01:55:54,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:54,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682560554657, max timestamp=1682560554696Num uniques 859; 2023-04-27 01:55:54,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:54,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:54,731 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,734 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,735 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,735 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560614729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560614730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,740 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,740 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,740 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,741 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,742 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,742 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,742 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:54,742 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:54,743 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560614734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560614735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:54,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b62849[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:54,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:54,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,746 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:54,746 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,746 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:54,746 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,748 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:54,750 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:54,751 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:54,751 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:54,751 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:54,751 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:54,761 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,761 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,762 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,762 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:54,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:54,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,767 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,767 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:54,767 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:54,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,837 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/ec27df05cb7c4de293d4f8682c4b47f0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/ec27df05cb7c4de293d4f8682c4b47f0 2023-04-27 01:55:54,841 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into ec27df05cb7c4de293d4f8682c4b47f0(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:54,841 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:54,841 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=11, startTime=1682560554409; duration=0sec 2023-04-27 01:55:54,841 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:54,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560614839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560614839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560614845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:54,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560614846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,868 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,871 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,871 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,873 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,873 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,873 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,873 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,874 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,874 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,874 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:54,875 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:54,877 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21746c8d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:54,878 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:54,878 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,879 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:54,880 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,880 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:54,882 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:54,884 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:54,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:54,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:54,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:54,884 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:54,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,885 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:54,886 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:54,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:54,887 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:54,890 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,890 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,890 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:54,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:54,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:54,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,890 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:54,890 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,890 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,890 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,891 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,891 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:54,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:54,890 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:54,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:54,891 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,891 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:54,991 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:54,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:54,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:54,994 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:54,994 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,994 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:54,994 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:54,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,996 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,996 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:54,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:54,996 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,997 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:54,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:54,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:54,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:54,998 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,998 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:54,998 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:54,998 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:55,000 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:55,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@685e80fa[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:55,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:55,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,003 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:55,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:55,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:55,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,006 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:55,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:55,009 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:55,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:55,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:55,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:55,009 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:55,010 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,010 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,010 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,011 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,015 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:55,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:55,016 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:55,016 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:55,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:55,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/6c32a06b23164467b63fd745c7aaa5ae 2023-04-27 01:55:55,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560615046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560615047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560615051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560615053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,116 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:55,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:55,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:55,119 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:55,119 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,119 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:55,119 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:55,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,120 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,121 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,121 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,122 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:55,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:55,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:55,123 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,123 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:55,123 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:55,123 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:55,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:55,129 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ab8622[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:55,131 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:55,131 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:55,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:55,134 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:55,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:55,137 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:55,137 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:55,137 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:55,137 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:55,137 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:55,143 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,143 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,143 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:55,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:55,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,149 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,148 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,149 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:55,149 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:55,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,249 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:55,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:55,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:55,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:55,253 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:55,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:55,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:55,254 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,254 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,255 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,255 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,255 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,255 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:55,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:55,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:55,257 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,257 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:55,257 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:55,257 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,259 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bdd113e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:55,261 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:55,261 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,268 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:55,268 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,268 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,270 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:55,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:55,272 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:55,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:55,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:55,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:55,273 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:55,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,278 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,279 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:55,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,281 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,282 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:55,282 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:55,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560615354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560615355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560615359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560615360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,382 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:55,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:55,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:55,385 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:55,385 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,385 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:55,385 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:55,385 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,386 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,388 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:55,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:55,390 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,390 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:55,390 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:55,390 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,391 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:55,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@194f16cc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:55,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:55,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,394 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:55,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:55,395 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,395 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:55,396 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:55,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:55,399 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:55,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:55,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:55,399 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:55,399 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:55,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,402 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,402 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,402 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:55,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:55,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:55,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:55,406 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,407 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:55,407 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:55,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/e46c6538bdaa4236be5f9ca30bca25ff 2023-04-27 01:55:55,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/95c588d5408b45cdbc59e68d94dcfde1 2023-04-27 01:55:55,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/6c32a06b23164467b63fd745c7aaa5ae as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/6c32a06b23164467b63fd745c7aaa5ae 2023-04-27 01:55:55,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/6c32a06b23164467b63fd745c7aaa5ae, entries=250, sequenceid=303, filesize=16.7 K 2023-04-27 01:55:55,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/e46c6538bdaa4236be5f9ca30bca25ff as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e46c6538bdaa4236be5f9ca30bca25ff 2023-04-27 01:55:55,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e46c6538bdaa4236be5f9ca30bca25ff, entries=150, sequenceid=303, filesize=11.9 K 2023-04-27 01:55:55,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/95c588d5408b45cdbc59e68d94dcfde1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/95c588d5408b45cdbc59e68d94dcfde1 2023-04-27 01:55:55,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/95c588d5408b45cdbc59e68d94dcfde1, entries=150, sequenceid=303, filesize=11.9 K 2023-04-27 01:55:55,499 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for bf86160e96ad9ed4e3ec62cf4c749648 in 870ms, sequenceid=303, compaction requested=true 2023-04-27 01:55:55,499 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:55,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:55,499 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:55,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:55,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 01:55:55,499 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:55,500 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41902 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:55,500 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:55:55,501 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,501 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/ec27df05cb7c4de293d4f8682c4b47f0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a557a1ffee3c4bba8b35136c13a09f48, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/6c32a06b23164467b63fd745c7aaa5ae] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=40.9 K 2023-04-27 01:55:55,501 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:55,501 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:55:55,501 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting ec27df05cb7c4de293d4f8682c4b47f0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682560553948 2023-04-27 01:55:55,501 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,501 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2538ad199cbb4e1aa2f0b336be278b7f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b6b24a59ef05413093fab6ee3ed276d5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e46c6538bdaa4236be5f9ca30bca25ff] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=36.2 K 2023-04-27 01:55:55,501 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting a557a1ffee3c4bba8b35136c13a09f48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1682560554044 2023-04-27 01:55:55,501 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2538ad199cbb4e1aa2f0b336be278b7f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682560553948 2023-04-27 01:55:55,502 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 6c32a06b23164467b63fd745c7aaa5ae, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682560554617 2023-04-27 01:55:55,502 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b6b24a59ef05413093fab6ee3ed276d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1682560554044 2023-04-27 01:55:55,502 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e46c6538bdaa4236be5f9ca30bca25ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682560554617 2023-04-27 01:55:55,507 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:55,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:55,509 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:55,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:55,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:55,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:55,512 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:55,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:55,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:55,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,513 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:55,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:55,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:55,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:55,517 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:55,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:55,532 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,532 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:55,532 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-27 01:55:55,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:55,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:55,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:55,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:55,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:55,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:55,555 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/fbca135e04e34fceafcb6cc3cb3e038d 2023-04-27 01:55:55,569 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/f7a528e3de20457aa069bce39ac69ff2 2023-04-27 01:55:55,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:55,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:55,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560615878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560615878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560615880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560615880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,931 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/4da29c656aa94b268253fdb158a22f1b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4da29c656aa94b268253fdb158a22f1b 2023-04-27 01:55:55,935 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into 4da29c656aa94b268253fdb158a22f1b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:55,935 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:55,935 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=13, startTime=1682560555499; duration=0sec 2023-04-27 01:55:55,935 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:55:55,935 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:55:55,936 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:55:55,936 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:55:55,936 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,936 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/84090761666b4393a8bab35774ca2fc2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b5064c7dbab546bab75b76fd8277b78a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/95c588d5408b45cdbc59e68d94dcfde1] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=36.2 K 2023-04-27 01:55:55,937 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 84090761666b4393a8bab35774ca2fc2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1682560553948 2023-04-27 01:55:55,937 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b5064c7dbab546bab75b76fd8277b78a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1682560554044 2023-04-27 01:55:55,937 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 95c588d5408b45cdbc59e68d94dcfde1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682560554617 2023-04-27 01:55:55,943 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:55:55,952 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/b805b000daf84dc9be3de41f93808766 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b805b000daf84dc9be3de41f93808766 2023-04-27 01:55:55,956 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/8528d83503104af38879579092cce630 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/8528d83503104af38879579092cce630 2023-04-27 01:55:55,956 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into b805b000daf84dc9be3de41f93808766(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:55,956 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:55,956 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=13, startTime=1682560555499; duration=0sec 2023-04-27 01:55:55,956 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:55,959 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 8528d83503104af38879579092cce630(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:55:55,959 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:55,959 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=13, startTime=1682560555499; duration=0sec 2023-04-27 01:55:55,959 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:55:55,979 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/4f93237789104965a5ce4944671d4735 2023-04-27 01:55:55,982 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/fbca135e04e34fceafcb6cc3cb3e038d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/fbca135e04e34fceafcb6cc3cb3e038d 2023-04-27 01:55:55,984 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/fbca135e04e34fceafcb6cc3cb3e038d, entries=150, sequenceid=339, filesize=11.9 K 2023-04-27 01:55:55,985 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/f7a528e3de20457aa069bce39ac69ff2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f7a528e3de20457aa069bce39ac69ff2 2023-04-27 01:55:55,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560615982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560615986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,989 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f7a528e3de20457aa069bce39ac69ff2, entries=150, sequenceid=339, filesize=11.9 K 2023-04-27 01:55:55,990 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/4f93237789104965a5ce4944671d4735 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4f93237789104965a5ce4944671d4735 2023-04-27 01:55:55,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560615987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560615988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,994 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4f93237789104965a5ce4944671d4735, entries=150, sequenceid=339, filesize=11.9 K 2023-04-27 01:55:55,995 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=53.67 KB/54960 for bf86160e96ad9ed4e3ec62cf4c749648 in 463ms, sequenceid=339, compaction requested=false 2023-04-27 01:55:55,995 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:55,995 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:55,997 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58a1c4f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:55,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:56,000 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,002 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:56,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:56,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:56,014 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:56,015 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:56,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:56,017 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:56,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:56,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:56,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:56,017 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:56,018 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,018 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:56,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:56,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:56,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:56,020 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:56,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:56,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:56,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:56,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:56,023 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:56,023 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:56,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:56,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:56,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:56,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:56,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:56,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,024 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,024 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:56,123 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:56,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:56,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:56,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:56,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:56,126 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:56,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:56,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:56,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:56,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:56,128 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:56,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:56,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,129 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:56,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:56,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:56,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:56,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:56,131 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:56,131 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:56,146 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/70b708a5fdce48eea7d6894b16d24c7b 2023-04-27 01:55:56,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:56,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:56,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560556212, max timestamp=1682560556221Num uniques 859; 2023-04-27 01:55:56,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:56,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560556212, max timestamp=1682560556221Num uniques 859; 2023-04-27 01:55:56,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682560556212, max timestamp=1682560556221Num uniques 859; 2023-04-27 01:55:56,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:56,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 01:55:56,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560616249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560616250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560616250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560616252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560616349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,354 DEBUG [Thread-2921] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:55:56,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560616359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560616359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560616359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560616360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,562 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/73f9085d84274e44aa22f9faaf1b7283 2023-04-27 01:55:56,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560616563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560616563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560616564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560616568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560616875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560616875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560616876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:56,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560616877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,971 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/fe7e5c4feff743f380107802bae9c4d6 2023-04-27 01:55:56,976 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/70b708a5fdce48eea7d6894b16d24c7b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/70b708a5fdce48eea7d6894b16d24c7b 2023-04-27 01:55:56,979 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/70b708a5fdce48eea7d6894b16d24c7b, entries=150, sequenceid=353, filesize=11.9 K 2023-04-27 01:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/73f9085d84274e44aa22f9faaf1b7283 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/73f9085d84274e44aa22f9faaf1b7283 2023-04-27 01:55:56,983 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/73f9085d84274e44aa22f9faaf1b7283, entries=150, sequenceid=353, filesize=11.9 K 2023-04-27 01:55:56,984 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/fe7e5c4feff743f380107802bae9c4d6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fe7e5c4feff743f380107802bae9c4d6 2023-04-27 01:55:56,987 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fe7e5c4feff743f380107802bae9c4d6, entries=150, sequenceid=353, filesize=11.9 K 2023-04-27 01:55:56,988 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bf86160e96ad9ed4e3ec62cf4c749648 in 857ms, sequenceid=353, compaction requested=true 2023-04-27 01:55:56,988 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:56,988 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:56,996 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4055004a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:56,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:56,998 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,999 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:56,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:56,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:57,000 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,000 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:57,003 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:57,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:57,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:57,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:57,003 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:57,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,004 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:57,005 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,007 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:57,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:57,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:57,012 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:57,013 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:57,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:57,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:57,113 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:57,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:57,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:57,116 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:57,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:57,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:57,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:57,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,116 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,117 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:57,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,118 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:57,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:57,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:57,119 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:57,119 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:57,119 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:57,120 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-27 01:55:57,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:57,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:57,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:57,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:57,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:57,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:57,126 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/69daad13ed3f4254aa1e362967c6f96a 2023-04-27 01:55:57,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:57,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:57,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560617399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560617400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560617401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560617401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560617510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560617510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560617510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560617510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,542 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/66529cdb6b594594a1410cd6fd0d0335 2023-04-27 01:55:57,552 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/3e04dda007084f4d85308b800f9dd00e 2023-04-27 01:55:57,556 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/69daad13ed3f4254aa1e362967c6f96a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/69daad13ed3f4254aa1e362967c6f96a 2023-04-27 01:55:57,559 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/69daad13ed3f4254aa1e362967c6f96a, entries=150, sequenceid=388, filesize=11.9 K 2023-04-27 01:55:57,560 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/66529cdb6b594594a1410cd6fd0d0335 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/66529cdb6b594594a1410cd6fd0d0335 2023-04-27 01:55:57,563 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/66529cdb6b594594a1410cd6fd0d0335, entries=150, sequenceid=388, filesize=11.9 K 2023-04-27 01:55:57,565 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/3e04dda007084f4d85308b800f9dd00e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3e04dda007084f4d85308b800f9dd00e 2023-04-27 01:55:57,568 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3e04dda007084f4d85308b800f9dd00e, entries=150, sequenceid=388, filesize=11.9 K 2023-04-27 01:55:57,569 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for bf86160e96ad9ed4e3ec62cf4c749648 in 449ms, sequenceid=388, compaction requested=true 2023-04-27 01:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:57,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3372ad9f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:57,573 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:57,573 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:57,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,574 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:57,577 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:57,577 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:57,580 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:57,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:57,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:57,581 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:57,581 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:57,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,582 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,583 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:57,585 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:57,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:57,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:57,587 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,587 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:57,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,587 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:57,587 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,687 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:57,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:57,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:57,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:57,691 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:57,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:57,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:57,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:57,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:57,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:57,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,693 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:57,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:57,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:57,694 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:57,694 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:57,694 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:55:57,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:57,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:57,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:57,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:57,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:57,695 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:57,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:57,721 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:55:57,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682560557747, max timestamp=1682560557762Num uniques 859; 2023-04-27 01:55:57,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:57,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682560557747, max timestamp=1682560557762Num uniques 859; 2023-04-27 01:55:57,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682560557747, max timestamp=1682560557762Num uniques 859; 2023-04-27 01:55:57,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:57,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 01:55:57,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560617792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560617796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560617797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560617798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560617899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560617900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560617905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:57,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:57,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560617907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,104 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/9a52bd5279ac4837850dd8725b4cc160 2023-04-27 01:55:58,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560618104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560618109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560618112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560618112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560618411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560618416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560618421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560618423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,517 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/51aa18297c184d8088545df84251abaa 2023-04-27 01:55:58,926 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/90242be229104372821add8272101ca9 2023-04-27 01:55:58,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560618921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560618922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,930 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/9a52bd5279ac4837850dd8725b4cc160 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9a52bd5279ac4837850dd8725b4cc160 2023-04-27 01:55:58,933 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9a52bd5279ac4837850dd8725b4cc160, entries=150, sequenceid=399, filesize=11.9 K 2023-04-27 01:55:58,934 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/51aa18297c184d8088545df84251abaa as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/51aa18297c184d8088545df84251abaa 2023-04-27 01:55:58,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560618933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,940 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/51aa18297c184d8088545df84251abaa, entries=150, sequenceid=399, filesize=11.9 K 2023-04-27 01:55:58,941 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/90242be229104372821add8272101ca9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/90242be229104372821add8272101ca9 2023-04-27 01:55:58,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:55:58,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560618935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,944 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/90242be229104372821add8272101ca9, entries=150, sequenceid=399, filesize=11.9 K 2023-04-27 01:55:58,945 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for bf86160e96ad9ed4e3ec62cf4c749648 in 1251ms, sequenceid=399, compaction requested=true 2023-04-27 01:55:58,945 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:58,945 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:58,947 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,947 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52ea344b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:58,949 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:58,949 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:58,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:58,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:58,956 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:58,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:58,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:58,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:58,956 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:58,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:58,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:58,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:58,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:58,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:58,960 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:58,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:58,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:58,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:58,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:58,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:58,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:58,965 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:58,965 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,065 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:59,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:59,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:59,068 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:59,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:59,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:59,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:59,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:59,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:59,074 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:59,074 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:55:59,085 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/3df98383918747cb921b04c6ecad09f8 2023-04-27 01:55:59,094 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/5e68592032564897866f85d4ae0cc82d 2023-04-27 01:55:59,103 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/b47c0889a92d4217a2b03810f78f2d1d 2023-04-27 01:55:59,117 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/3df98383918747cb921b04c6ecad09f8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3df98383918747cb921b04c6ecad09f8 2023-04-27 01:55:59,120 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3df98383918747cb921b04c6ecad09f8, entries=150, sequenceid=433, filesize=11.9 K 2023-04-27 01:55:59,121 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/5e68592032564897866f85d4ae0cc82d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/5e68592032564897866f85d4ae0cc82d 2023-04-27 01:55:59,124 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/5e68592032564897866f85d4ae0cc82d, entries=150, sequenceid=433, filesize=11.9 K 2023-04-27 01:55:59,124 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/b47c0889a92d4217a2b03810f78f2d1d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b47c0889a92d4217a2b03810f78f2d1d 2023-04-27 01:55:59,127 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b47c0889a92d4217a2b03810f78f2d1d, entries=150, sequenceid=433, filesize=11.9 K 2023-04-27 01:55:59,128 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=0 B/0 for bf86160e96ad9ed4e3ec62cf4c749648 in 54ms, sequenceid=433, compaction requested=true 2023-04-27 01:55:59,128 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:59,128 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:59,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:59,130 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:59,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757223d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:59,132 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:59,132 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,133 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,133 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:59,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:59,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:59,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,135 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:59,137 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:59,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:59,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:59,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:59,138 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:59,140 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,140 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,140 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,141 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:59,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:59,146 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:59,146 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,147 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,147 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,246 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:59,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:59,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:59,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:59,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:59,250 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:59,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:59,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:59,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,252 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,252 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:59,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:59,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:59,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:59,254 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,254 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:59,254 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:59,254 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:59,255 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,255 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6825cd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:59,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:59,257 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,259 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:59,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:59,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:59,261 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:59,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,261 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:59,263 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:59,263 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:59,263 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:59,263 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:59,263 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:59,266 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,266 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,266 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,266 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:59,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:59,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:59,272 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:59,272 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,373 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:59,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:59,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:59,376 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:59,376 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:59,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:59,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,384 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,385 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,385 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:59,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:59,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:59,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:59,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:59,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:59,386 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:59,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:59,388 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@495063ea[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:59,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:59,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,393 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,393 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:59,393 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:59,393 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:59,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:59,395 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:59,398 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:59,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:59,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:59,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:59,398 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:59,402 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,402 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,402 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,404 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:59,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:59,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,410 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:59,410 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,410 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:59,511 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:59,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:59,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:59,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:59,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:59,514 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:59,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:59,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:59,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,514 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,515 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,516 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:59,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:59,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:59,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:59,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:59,519 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:59,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:59,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:59,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a9b4ee7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:59,524 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:59,524 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:59,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:59,525 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,527 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:59,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:59,529 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:59,529 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:59,529 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:59,529 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:59,529 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:59,530 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,530 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,530 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,533 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,536 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:59,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,537 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:59,537 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,537 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,637 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:59,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:59,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:59,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:59,640 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:59,640 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,640 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:59,640 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:59,641 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,641 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,642 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,642 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,643 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,643 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:59,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:59,644 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:59,644 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,644 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:59,645 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:59,645 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:59,645 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,646 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:59,647 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:59,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7463db43[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:59,649 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:59,649 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,650 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,650 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,650 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:59,650 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:59,650 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:59,650 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:59,653 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,653 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:59,653 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,653 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:59,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:59,655 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:59,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:59,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:59,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:59,655 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:59,657 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,657 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,657 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,658 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,659 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,661 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:59,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:59,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,662 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,662 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:59,662 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,762 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:59,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:59,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:59,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:59,766 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:59,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:59,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:59,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,767 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,768 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,768 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:59,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:59,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:59,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:59,770 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:59,773 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,773 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0bde4f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:59,775 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:59,775 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,776 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:59,776 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,776 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:59,778 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:59,778 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,778 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,778 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:59,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:59,782 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:59,782 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:59,782 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:59,782 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:59,782 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:59,783 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,783 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,783 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,784 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:59,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:59,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:59,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,791 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:59,791 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,891 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:55:59,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:55:59,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:55:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:55:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:55:59,894 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:55:59,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:55:59,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:55:59,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,896 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:55:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:55:59,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:55:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:55:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:55:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:55:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:55:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:55:59,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:55:59,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:55:59,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:55:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:55:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:55:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:55:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:55:59,899 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:55:59,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108ea9d7[Count = 0] remaining members to acquire global barrier 2023-04-27 01:55:59,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:55:59,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:55:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:55:59,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:55:59,903 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,903 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:55:59,906 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:55:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:55:59,908 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:55:59,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:55:59,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:55:59,908 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:55:59,908 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:55:59,910 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,910 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:55:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:55:59,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,910 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:55:59,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:55:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:55:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:55:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:55:59,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:55:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:55:59,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:55:59,914 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:55:59,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:55:59,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:55:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:55:59,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:55:59,915 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:55:59,915 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:55:59,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:55:59,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:55:59,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:55:59,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:59,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:55:59,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:59,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:55:59,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:55:59,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682560559974, max timestamp=1682560559995Num uniques 859; 2023-04-27 01:55:59,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 01:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682560559974, max timestamp=1682560559995Num uniques 859; 2023-04-27 01:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 01:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682560559974, max timestamp=1682560559995Num uniques 859; 2023-04-27 01:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 01:56:00,015 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,019 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,020 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,020 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,021 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,022 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,022 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,022 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:00,022 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:00,024 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,024 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,025 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,025 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:00,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722157[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:00,026 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:00,026 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,027 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:00,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:00,028 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:00,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:00,032 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:00,032 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:00,032 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:00,032 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:00,032 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:00,034 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,034 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,034 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,035 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560620030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,037 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,040 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:00,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,040 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:00,040 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:00,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:00,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560620033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560620035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560620036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,140 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,143 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560620137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,147 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,147 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,148 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,148 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,149 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,149 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,149 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:00,149 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:00,150 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:00,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa14bd9[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:00,152 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:00,152 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,153 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:00,154 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,154 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:00,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,156 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:00,158 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:00,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:00,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:00,159 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:00,159 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:00,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,161 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,161 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,162 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560620148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560620148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560620154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,165 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,165 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:00,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:00,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,165 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:00,166 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,266 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,269 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,270 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,270 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,271 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,273 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,273 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,273 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:00,273 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:00,278 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,279 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,279 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e5c0964[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:00,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:00,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:00,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,286 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:00,288 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:00,288 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:00,288 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:00,288 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:00,288 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:00,290 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,290 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,290 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,295 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:00,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:00,298 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,298 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,298 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,298 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3 2023-04-27 01:56:00,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560620347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46430 deadline: 1682560620357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,370 DEBUG [Thread-2921] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:00,373 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/fa2979b1189c4f2d86257f098aeecadb 2023-04-27 01:56:00,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560620365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560620365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560620365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,399 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,403 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,404 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,405 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,405 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,406 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,406 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,406 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:00,406 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:00,408 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b79c569[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,414 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:00,414 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:00,414 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,414 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,415 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:00,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:00,418 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:00,418 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:00,418 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:00,418 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:00,418 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:00,419 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,420 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,420 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,420 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,423 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:00,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:00,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,423 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,424 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:00,424 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:00,424 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,524 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,527 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,528 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,531 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:00,533 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:00,536 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:00,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1025c8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:00,538 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:00,538 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,539 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,539 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:00,539 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:00,539 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,539 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:00,539 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:00,541 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,541 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:00,541 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,541 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:00,543 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:00,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:00,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:00,543 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:00,544 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:00,546 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,546 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,546 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,549 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,551 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,551 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,551 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:00,552 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:00,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,652 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,655 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,655 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,656 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,657 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,657 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,657 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,657 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:00,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,661 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,661 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,661 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0f5ad0[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:00,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:00,662 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,664 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:00,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:00,664 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:00,665 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,665 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:00,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,666 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:00,668 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:00,668 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:00,668 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:00,668 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:00,668 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:00,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,669 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:00,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:00,674 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,675 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,674 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,675 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,675 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,675 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:00,675 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,675 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:00,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560620670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560620678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560620678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560620679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,775 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,778 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,778 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,779 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/dcc765d22cfd4c2f8f42b409aa823dd5 2023-04-27 01:56:00,783 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,783 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,784 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,784 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3 2023-04-27 01:56:00,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,787 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,787 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,787 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:00,787 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,788 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3, entries=200, sequenceid=449, filesize=14.3 K 2023-04-27 01:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/fa2979b1189c4f2d86257f098aeecadb as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/fa2979b1189c4f2d86257f098aeecadb 2023-04-27 01:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74167bdc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:00,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:00,791 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,792 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:00,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:00,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:00,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:00,794 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:00,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,794 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,794 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/fa2979b1189c4f2d86257f098aeecadb, entries=150, sequenceid=449, filesize=11.9 K 2023-04-27 01:56:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/dcc765d22cfd4c2f8f42b409aa823dd5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/dcc765d22cfd4c2f8f42b409aa823dd5 2023-04-27 01:56:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:00,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:00,798 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:00,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:00,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:00,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:00,798 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:00,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/dcc765d22cfd4c2f8f42b409aa823dd5, entries=150, sequenceid=449, filesize=11.9 K 2023-04-27 01:56:00,800 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,800 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bf86160e96ad9ed4e3ec62cf4c749648 in 850ms, sequenceid=449, compaction requested=true 2023-04-27 01:56:00,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,800 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:00,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:00,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,801 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 01:56:00,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:00,801 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 01:56:00,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,803 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88458 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 01:56:00,803 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86018 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 01:56:00,803 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:56:00,803 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:56:00,803 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,803 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,803 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4da29c656aa94b268253fdb158a22f1b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/fbca135e04e34fceafcb6cc3cb3e038d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/70b708a5fdce48eea7d6894b16d24c7b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/69daad13ed3f4254aa1e362967c6f96a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9a52bd5279ac4837850dd8725b4cc160, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3df98383918747cb921b04c6ecad09f8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=86.4 K 2023-04-27 01:56:00,803 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b805b000daf84dc9be3de41f93808766, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f7a528e3de20457aa069bce39ac69ff2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/73f9085d84274e44aa22f9faaf1b7283, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/66529cdb6b594594a1410cd6fd0d0335, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/51aa18297c184d8088545df84251abaa, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/5e68592032564897866f85d4ae0cc82d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/fa2979b1189c4f2d86257f098aeecadb] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=84.0 K 2023-04-27 01:56:00,804 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 4da29c656aa94b268253fdb158a22f1b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682560554617 2023-04-27 01:56:00,804 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b805b000daf84dc9be3de41f93808766, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682560554617 2023-04-27 01:56:00,804 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting fbca135e04e34fceafcb6cc3cb3e038d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1682560554723 2023-04-27 01:56:00,804 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f7a528e3de20457aa069bce39ac69ff2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1682560554723 2023-04-27 01:56:00,804 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 70b708a5fdce48eea7d6894b16d24c7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682560555869 2023-04-27 01:56:00,805 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 73f9085d84274e44aa22f9faaf1b7283, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682560555869 2023-04-27 01:56:00,805 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 69daad13ed3f4254aa1e362967c6f96a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1682560556239 2023-04-27 01:56:00,805 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 66529cdb6b594594a1410cd6fd0d0335, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1682560556239 2023-04-27 01:56:00,805 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9a52bd5279ac4837850dd8725b4cc160, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682560557388 2023-04-27 01:56:00,805 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 51aa18297c184d8088545df84251abaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682560557388 2023-04-27 01:56:00,805 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3df98383918747cb921b04c6ecad09f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682560557795 2023-04-27 01:56:00,806 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cdd4d66116bc41c3a1ef73ee1cbbfbc3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682560559944 2023-04-27 01:56:00,806 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 5e68592032564897866f85d4ae0cc82d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682560557795 2023-04-27 01:56:00,806 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting fa2979b1189c4f2d86257f098aeecadb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682560559944 2023-04-27 01:56:00,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:00,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:00,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:00,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,814 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:00,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:00,815 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:00,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:00,815 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:00,820 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#462 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:00,822 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#463 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:00,916 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:00,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:00,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:00,919 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:00,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:00,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:00,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,919 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,920 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,921 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:00,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:00,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:00,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:00,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:00,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:00,922 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:00,923 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-27 01:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:01,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:01,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:01,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560621198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560621211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560621212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560621213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,235 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/86f63ed9614f4dc9843cdc7b43dd7d8f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/86f63ed9614f4dc9843cdc7b43dd7d8f 2023-04-27 01:56:01,240 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/0dfa10ae61844534977c297c8fc117db as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0dfa10ae61844534977c297c8fc117db 2023-04-27 01:56:01,244 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into 86f63ed9614f4dc9843cdc7b43dd7d8f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:01,244 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:01,244 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=9, startTime=1682560560801; duration=0sec 2023-04-27 01:56:01,244 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:01,244 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 01:56:01,245 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into 0dfa10ae61844534977c297c8fc117db(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:01,245 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:01,245 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=9, startTime=1682560560801; duration=0sec 2023-04-27 01:56:01,245 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:01,246 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86018 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 01:56:01,247 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:56:01,247 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:01,247 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/8528d83503104af38879579092cce630, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4f93237789104965a5ce4944671d4735, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fe7e5c4feff743f380107802bae9c4d6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3e04dda007084f4d85308b800f9dd00e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/90242be229104372821add8272101ca9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b47c0889a92d4217a2b03810f78f2d1d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/dcc765d22cfd4c2f8f42b409aa823dd5] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=84.0 K 2023-04-27 01:56:01,247 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8528d83503104af38879579092cce630, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682560554617 2023-04-27 01:56:01,247 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 4f93237789104965a5ce4944671d4735, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1682560554723 2023-04-27 01:56:01,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting fe7e5c4feff743f380107802bae9c4d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682560555869 2023-04-27 01:56:01,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3e04dda007084f4d85308b800f9dd00e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1682560556239 2023-04-27 01:56:01,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 90242be229104372821add8272101ca9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682560557388 2023-04-27 01:56:01,249 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b47c0889a92d4217a2b03810f78f2d1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682560557795 2023-04-27 01:56:01,249 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting dcc765d22cfd4c2f8f42b409aa823dd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682560559944 2023-04-27 01:56:01,261 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#465 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:01,270 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/b495461c85df498f8cb5b84cf03b24f5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b495461c85df498f8cb5b84cf03b24f5 2023-04-27 01:56:01,277 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into b495461c85df498f8cb5b84cf03b24f5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:01,277 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:01,277 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=9, startTime=1682560560801; duration=0sec 2023-04-27 01:56:01,277 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560621314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560621324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560621324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560621324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,334 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/1f41b66a965945d88b22abb484d97ec6 2023-04-27 01:56:01,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560621519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560621528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560621528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560621529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,758 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/d9d9b41647fb4c3dbd3d8c42ee31888d 2023-04-27 01:56:01,769 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/32014020d82b4d51a670d34d64760583 2023-04-27 01:56:01,773 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/1f41b66a965945d88b22abb484d97ec6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/1f41b66a965945d88b22abb484d97ec6 2023-04-27 01:56:01,777 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/1f41b66a965945d88b22abb484d97ec6, entries=150, sequenceid=484, filesize=11.9 K 2023-04-27 01:56:01,778 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/d9d9b41647fb4c3dbd3d8c42ee31888d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9d9b41647fb4c3dbd3d8c42ee31888d 2023-04-27 01:56:01,783 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9d9b41647fb4c3dbd3d8c42ee31888d, entries=150, sequenceid=484, filesize=11.9 K 2023-04-27 01:56:01,784 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/32014020d82b4d51a670d34d64760583 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/32014020d82b4d51a670d34d64760583 2023-04-27 01:56:01,787 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/32014020d82b4d51a670d34d64760583, entries=150, sequenceid=484, filesize=11.9 K 2023-04-27 01:56:01,788 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for bf86160e96ad9ed4e3ec62cf4c749648 in 865ms, sequenceid=484, compaction requested=false 2023-04-27 01:56:01,788 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:01,788 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:01,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:01,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,792 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3181baba[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:01,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:01,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,797 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,797 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:01,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:01,798 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:01,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:01,801 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:01,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:01,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:01,801 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:01,801 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:01,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:01,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:01,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:01,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,802 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:01,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,806 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:01,808 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:01,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:01,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:01,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:01,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,809 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,808 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:01,809 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:01,809 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:01,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:01,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:01,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:56:01,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:01,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:01,830 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:01,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:01,830 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:01,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:01,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560561858, max timestamp=1682560561865Num uniques 859; 2023-04-27 01:56:01,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:56:01,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560561858, max timestamp=1682560561865Num uniques 859; 2023-04-27 01:56:01,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:56:01,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682560561858, max timestamp=1682560561865Num uniques 859; 2023-04-27 01:56:01,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 01:56:01,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560621892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560621893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:01,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560621897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560621894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,909 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:01,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:01,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:01,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:01,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:01,912 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:01,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:01,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:01,912 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:01,913 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,913 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:01,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,915 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:01,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:01,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:01,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:01,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:01,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:01,916 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:01,917 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:01,917 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:01,919 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,919 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:01,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:01,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a24e727[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:01,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:01,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,922 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:01,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:01,924 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:01,927 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:01,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:01,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:01,927 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:01,927 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:01,931 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,931 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:01,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:01,932 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,932 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:01,933 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,935 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:01,935 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,935 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:01,935 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:01,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:01,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:01,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:01,935 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:01,935 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:01,935 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:01,935 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:01,935 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:01,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:01,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:01,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:01,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:02,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560622002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560622004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560622004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560622004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,036 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:02,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:02,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:02,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:02,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:02,039 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:02,039 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:02,039 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:02,039 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:02,040 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,040 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,041 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:02,041 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:02,042 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,042 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:02,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:02,043 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,043 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:02,043 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:02,043 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:02,044 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:02,045 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a7df169[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:02,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:02,058 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,060 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:02,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:02,060 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:02,060 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,061 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:02,062 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:02,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:02,065 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:02,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:02,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:02,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:02,065 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:02,067 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,067 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:02,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,067 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:02,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:02,067 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:02,071 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:02,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:02,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:02,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:02,071 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:02,072 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:02,072 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:02,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:02,172 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:02,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:02,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:02,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:02,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:02,175 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:02,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:02,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:02,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:02,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,178 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:02,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:02,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:02,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:02,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:02,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:02,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:02,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:02,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,181 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:02,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:02,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df2a761[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:02,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:02,183 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,184 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:02,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:02,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:02,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:02,189 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:02,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:02,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:02,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:02,189 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:02,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:02,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:02,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:02,194 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:02,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,196 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:02,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:02,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:02,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:02,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:02,197 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:02,197 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:02,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560622209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560622210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560622211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560622212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,238 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/9202a3f8285941cda2da5ab6f0907a57 2023-04-27 01:56:02,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/3b3eb2315dfd4bcbb7fb471992f0e01d 2023-04-27 01:56:02,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/bbaa1daf05944efa8ef08e9ff748a50b 2023-04-27 01:56:02,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/9202a3f8285941cda2da5ab6f0907a57 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9202a3f8285941cda2da5ab6f0907a57 2023-04-27 01:56:02,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9202a3f8285941cda2da5ab6f0907a57, entries=150, sequenceid=498, filesize=11.9 K 2023-04-27 01:56:02,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/3b3eb2315dfd4bcbb7fb471992f0e01d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3b3eb2315dfd4bcbb7fb471992f0e01d 2023-04-27 01:56:02,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3b3eb2315dfd4bcbb7fb471992f0e01d, entries=150, sequenceid=498, filesize=11.9 K 2023-04-27 01:56:02,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/bbaa1daf05944efa8ef08e9ff748a50b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/bbaa1daf05944efa8ef08e9ff748a50b 2023-04-27 01:56:02,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/bbaa1daf05944efa8ef08e9ff748a50b, entries=150, sequenceid=498, filesize=11.9 K 2023-04-27 01:56:02,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bf86160e96ad9ed4e3ec62cf4c749648 in 447ms, sequenceid=498, compaction requested=true 2023-04-27 01:56:02,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:02,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:02,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:02,277 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:02,277 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:02,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:02,278 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:02,278 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:02,278 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:56:02,278 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:56:02,278 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,278 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,278 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/86f63ed9614f4dc9843cdc7b43dd7d8f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9d9b41647fb4c3dbd3d8c42ee31888d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3b3eb2315dfd4bcbb7fb471992f0e01d] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=36.6 K 2023-04-27 01:56:02,278 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0dfa10ae61844534977c297c8fc117db, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/1f41b66a965945d88b22abb484d97ec6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9202a3f8285941cda2da5ab6f0907a57] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=36.6 K 2023-04-27 01:56:02,278 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 86f63ed9614f4dc9843cdc7b43dd7d8f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682560559944 2023-04-27 01:56:02,279 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 0dfa10ae61844534977c297c8fc117db, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682560559944 2023-04-27 01:56:02,279 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d9d9b41647fb4c3dbd3d8c42ee31888d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682560560020 2023-04-27 01:56:02,279 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 1f41b66a965945d88b22abb484d97ec6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682560560020 2023-04-27 01:56:02,279 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3b3eb2315dfd4bcbb7fb471992f0e01d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682560561197 2023-04-27 01:56:02,279 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9202a3f8285941cda2da5ab6f0907a57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682560561197 2023-04-27 01:56:02,286 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:02,287 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#472 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:02,297 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:02,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:02,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:02,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:02,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:02,301 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:02,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:02,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:02,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:02,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:02,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:02,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:02,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:02,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:02,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:02,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:02,305 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-27 01:56:02,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:02,306 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:02,306 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:02,306 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:02,306 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:02,306 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:02,314 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/90f5985cb1c34f3ca6b2886c0405e75d 2023-04-27 01:56:02,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:02,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:02,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560622531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560622532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560622536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560622540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560622641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560622641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560622647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560622649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,700 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/ae8a1f3b307744c79dd1d60eb205c68c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ae8a1f3b307744c79dd1d60eb205c68c 2023-04-27 01:56:02,701 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/5787e7bd246e44b0ba0a38d57ff49a64 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5787e7bd246e44b0ba0a38d57ff49a64 2023-04-27 01:56:02,704 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into ae8a1f3b307744c79dd1d60eb205c68c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:02,704 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:02,704 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=13, startTime=1682560562277; duration=0sec 2023-04-27 01:56:02,704 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:02,704 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:02,705 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into 5787e7bd246e44b0ba0a38d57ff49a64(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:02,705 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:02,705 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=13, startTime=1682560562277; duration=0sec 2023-04-27 01:56:02,705 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:02,705 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:02,705 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:56:02,705 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:02,705 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b495461c85df498f8cb5b84cf03b24f5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/32014020d82b4d51a670d34d64760583, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/bbaa1daf05944efa8ef08e9ff748a50b] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=36.6 K 2023-04-27 01:56:02,706 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b495461c85df498f8cb5b84cf03b24f5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682560559944 2023-04-27 01:56:02,706 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 32014020d82b4d51a670d34d64760583, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682560560020 2023-04-27 01:56:02,706 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting bbaa1daf05944efa8ef08e9ff748a50b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682560561197 2023-04-27 01:56:02,711 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:02,726 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/12d327e3583a4796beda701170370351 2023-04-27 01:56:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560622851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560622852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560622852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:02,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560622854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,122 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/51bf0cf1ed7d474cb77db50de0ff5a5b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/51bf0cf1ed7d474cb77db50de0ff5a5b 2023-04-27 01:56:03,126 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 51bf0cf1ed7d474cb77db50de0ff5a5b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:03,126 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:03,126 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=13, startTime=1682560562277; duration=0sec 2023-04-27 01:56:03,127 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:03,136 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/f6d10b5fc38d4cfdab78fbe711976cff 2023-04-27 01:56:03,140 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/90f5985cb1c34f3ca6b2886c0405e75d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/90f5985cb1c34f3ca6b2886c0405e75d 2023-04-27 01:56:03,145 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/90f5985cb1c34f3ca6b2886c0405e75d, entries=150, sequenceid=534, filesize=11.9 K 2023-04-27 01:56:03,145 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/12d327e3583a4796beda701170370351 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/12d327e3583a4796beda701170370351 2023-04-27 01:56:03,149 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/12d327e3583a4796beda701170370351, entries=150, sequenceid=534, filesize=11.9 K 2023-04-27 01:56:03,150 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/f6d10b5fc38d4cfdab78fbe711976cff as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f6d10b5fc38d4cfdab78fbe711976cff 2023-04-27 01:56:03,154 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f6d10b5fc38d4cfdab78fbe711976cff, entries=150, sequenceid=534, filesize=11.9 K 2023-04-27 01:56:03,155 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for bf86160e96ad9ed4e3ec62cf4c749648 in 850ms, sequenceid=534, compaction requested=false 2023-04-27 01:56:03,155 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:03,155 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:03,157 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,157 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:03,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:56:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:03,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:03,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519a940f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:03,163 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:03,163 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,165 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,165 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:03,166 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,166 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:03,166 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:03,166 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:03,167 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:03,167 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,167 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:03,168 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,170 WARN [DataStreamer for file /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/3930737e22164f22945d4a597bd92940] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-27 01:56:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/3930737e22164f22945d4a597bd92940 2023-04-27 01:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:03,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:03,173 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:03,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:03,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:03,173 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:03,173 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:03,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,176 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,178 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,185 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:03,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:03,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,185 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:03,186 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:03,185 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,186 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/e0c07f56149545f29287328bbad31c82 2023-04-27 01:56:03,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560623183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560623183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560623185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560623190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,286 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:03,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:03,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:03,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:03,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:03,289 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:03,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:03,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:03,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,290 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,291 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,291 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,292 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:03,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:03,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:03,293 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:03,293 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,293 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:03,295 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,296 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:03,297 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:03,297 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:03,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:03,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:03,297 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:03,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,299 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,299 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0a5cc4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:03,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:03,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560623296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560623296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560623296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560623297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:03,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:03,302 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:03,309 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:03,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,309 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:03,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:03,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:03,310 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:03,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:03,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:03,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:03,310 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:03,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,312 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,316 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,316 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,316 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:03,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:03,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,316 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:03,316 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:03,316 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,417 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:03,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:03,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:03,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:03,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:03,420 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:03,420 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,420 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:03,420 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:03,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,421 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,422 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,422 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:03,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:03,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:03,425 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:03,425 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,425 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:03,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:03,426 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:03,426 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:03,430 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,430 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:03,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19390895[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:03,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:03,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,433 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:03,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:03,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,433 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,434 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:03,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:03,437 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:03,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:03,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:03,437 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:03,437 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:03,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,439 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:03,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,443 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,444 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,443 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,444 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:03,444 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:03,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:03,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560623502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560623502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560623503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560623503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,544 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:03,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:03,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:03,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:03,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:03,547 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:03,547 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:03,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:03,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,548 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,552 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,552 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,552 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,552 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:03,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:03,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:03,554 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,554 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:03,554 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:03,554 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:03,556 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,556 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:03,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61086d7c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:03,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:03,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:03,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:03,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:03,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,560 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:03,561 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:03,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:03,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:03,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:03,563 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:03,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:03,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:03,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:03,563 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:03,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,566 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:03,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:03,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:03,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:03,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:03,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:03,573 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:03,574 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:03,573 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:03,574 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:03,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:03,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/2c92de635ea84e6fbb190a7be07396d9 2023-04-27 01:56:03,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/3930737e22164f22945d4a597bd92940 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3930737e22164f22945d4a597bd92940 2023-04-27 01:56:03,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3930737e22164f22945d4a597bd92940, entries=200, sequenceid=552, filesize=14.3 K 2023-04-27 01:56:03,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/e0c07f56149545f29287328bbad31c82 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e0c07f56149545f29287328bbad31c82 2023-04-27 01:56:03,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e0c07f56149545f29287328bbad31c82, entries=150, sequenceid=552, filesize=11.9 K 2023-04-27 01:56:03,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/2c92de635ea84e6fbb190a7be07396d9 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2c92de635ea84e6fbb190a7be07396d9 2023-04-27 01:56:03,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2c92de635ea84e6fbb190a7be07396d9, entries=150, sequenceid=552, filesize=11.9 K 2023-04-27 01:56:03,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for bf86160e96ad9ed4e3ec62cf4c749648 in 468ms, sequenceid=552, compaction requested=true 2023-04-27 01:56:03,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:03,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:03,629 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:03,629 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:03,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:03,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:03,630 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:03,630 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:56:03,630 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,630 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5787e7bd246e44b0ba0a38d57ff49a64, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/90f5985cb1c34f3ca6b2886c0405e75d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3930737e22164f22945d4a597bd92940] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=39.1 K 2023-04-27 01:56:03,631 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:03,631 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 5787e7bd246e44b0ba0a38d57ff49a64, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682560561197 2023-04-27 01:56:03,631 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:56:03,631 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,631 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ae8a1f3b307744c79dd1d60eb205c68c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/12d327e3583a4796beda701170370351, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e0c07f56149545f29287328bbad31c82] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=36.7 K 2023-04-27 01:56:03,632 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 90f5985cb1c34f3ca6b2886c0405e75d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682560561890 2023-04-27 01:56:03,632 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ae8a1f3b307744c79dd1d60eb205c68c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682560561197 2023-04-27 01:56:03,632 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 3930737e22164f22945d4a597bd92940, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682560563159 2023-04-27 01:56:03,632 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 12d327e3583a4796beda701170370351, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682560561890 2023-04-27 01:56:03,633 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e0c07f56149545f29287328bbad31c82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682560563159 2023-04-27 01:56:03,641 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:03,653 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:03,673 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/cb8ad60504504d6b9c7fa4ad17acb475 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cb8ad60504504d6b9c7fa4ad17acb475 2023-04-27 01:56:03,674 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:03,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:03,677 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into cb8ad60504504d6b9c7fa4ad17acb475(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:03,677 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:03,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:03,677 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=13, startTime=1682560563628; duration=0sec 2023-04-27 01:56:03,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:03,677 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:03,677 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:03,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,677 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:03,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:03,677 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:03,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,678 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,678 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:03,678 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:56:03,678 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,678 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/51bf0cf1ed7d474cb77db50de0ff5a5b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f6d10b5fc38d4cfdab78fbe711976cff, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2c92de635ea84e6fbb190a7be07396d9] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=36.7 K 2023-04-27 01:56:03,679 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 51bf0cf1ed7d474cb77db50de0ff5a5b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682560561197 2023-04-27 01:56:03,679 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:03,679 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f6d10b5fc38d4cfdab78fbe711976cff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682560561890 2023-04-27 01:56:03,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:03,679 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2c92de635ea84e6fbb190a7be07396d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682560563159 2023-04-27 01:56:03,679 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,680 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:03,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:03,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:03,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:03,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:03,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:03,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:03,681 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:03,681 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:03,681 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:03,690 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:03,715 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/7a3a8a7b3846475fa8f4765dae460e28 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/7a3a8a7b3846475fa8f4765dae460e28 2023-04-27 01:56:03,719 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 7a3a8a7b3846475fa8f4765dae460e28(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:03,719 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:03,719 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=13, startTime=1682560563629; duration=0sec 2023-04-27 01:56:03,719 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:03,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:03,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:03,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560623828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560623829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560623829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560623830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560623935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560623936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560623937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:03,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:03,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560623937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,075 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/3ce6ae80db664cec94c21b991be1c19f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3ce6ae80db664cec94c21b991be1c19f 2023-04-27 01:56:04,079 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into 3ce6ae80db664cec94c21b991be1c19f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:04,079 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:04,079 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=13, startTime=1682560563629; duration=0sec 2023-04-27 01:56:04,079 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:04,104 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/5d3999597bd74922b9803102c0bb05c1 2023-04-27 01:56:04,119 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/2218a3eb443747938bf5bbfdc7f757e7 2023-04-27 01:56:04,138 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/aeba0c64ea014b4a9117c46a528447c7 2023-04-27 01:56:04,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560624139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560624142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560624143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560624143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,146 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/5d3999597bd74922b9803102c0bb05c1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5d3999597bd74922b9803102c0bb05c1 2023-04-27 01:56:04,150 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5d3999597bd74922b9803102c0bb05c1, entries=150, sequenceid=570, filesize=11.9 K 2023-04-27 01:56:04,151 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/2218a3eb443747938bf5bbfdc7f757e7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2218a3eb443747938bf5bbfdc7f757e7 2023-04-27 01:56:04,157 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2218a3eb443747938bf5bbfdc7f757e7, entries=150, sequenceid=570, filesize=11.9 K 2023-04-27 01:56:04,158 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/aeba0c64ea014b4a9117c46a528447c7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/aeba0c64ea014b4a9117c46a528447c7 2023-04-27 01:56:04,169 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/aeba0c64ea014b4a9117c46a528447c7, entries=150, sequenceid=570, filesize=11.9 K 2023-04-27 01:56:04,169 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for bf86160e96ad9ed4e3ec62cf4c749648 in 488ms, sequenceid=570, compaction requested=false 2023-04-27 01:56:04,169 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:04,170 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:04,171 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:04,172 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b7a7be[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:04,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:04,174 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,175 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:04,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,175 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:04,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:04,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:04,177 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,177 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:04,184 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:04,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:04,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:04,184 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:04,184 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:04,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:04,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:04,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:04,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:04,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:04,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:04,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:04,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:04,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:04,193 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:04,193 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:04,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:04,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:04,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:04,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:04,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:04,293 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:04,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:04,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:04,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:04,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:04,300 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:04,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:04,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:04,300 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:04,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,301 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:04,302 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:04,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:04,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:04,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:04,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:04,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:04,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:04,304 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:04,304 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. started... 2023-04-27 01:56:04,304 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 01:56:04,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:04,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:04,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:04,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:04,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:04,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:04,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:04,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. as already flushing 2023-04-27 01:56:04,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560624463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560624465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560624467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560624468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560624569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560624573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560624574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560624574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,720 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/f6eae87f24f1439b9e4d51ac0f4219f1 2023-04-27 01:56:04,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560624774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560624779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560624779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:04,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560624779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:04,793 DEBUG [Thread-2932] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1401cac1 to 127.0.0.1:52731 2023-04-27 01:56:04,793 DEBUG [Thread-2932] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:04,798 DEBUG [Thread-2928] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f65627a to 127.0.0.1:52731 2023-04-27 01:56:04,798 DEBUG [Thread-2928] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:04,798 DEBUG [Thread-2934] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f6e865f to 127.0.0.1:52731 2023-04-27 01:56:04,798 DEBUG [Thread-2934] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:04,800 DEBUG [Thread-2930] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15f74c78 to 127.0.0.1:52731 2023-04-27 01:56:04,800 DEBUG [Thread-2930] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:04,803 DEBUG [Thread-2936] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48c22118 to 127.0.0.1:52731 2023-04-27 01:56:04,803 DEBUG [Thread-2936] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:05,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:05,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46404 deadline: 1682560625083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46424 deadline: 1682560625084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:05,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:05,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46414 deadline: 1682560625087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46420 deadline: 1682560625087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,130 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/d75b586fc4914bbe9fd71dc1ededee16 2023-04-27 01:56:05,541 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/ce8deaefcb5641d5bc0c7a72ba2b6370 2023-04-27 01:56:05,545 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/f6eae87f24f1439b9e4d51ac0f4219f1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f6eae87f24f1439b9e4d51ac0f4219f1 2023-04-27 01:56:05,547 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f6eae87f24f1439b9e4d51ac0f4219f1, entries=150, sequenceid=588, filesize=11.9 K 2023-04-27 01:56:05,548 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/d75b586fc4914bbe9fd71dc1ededee16 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d75b586fc4914bbe9fd71dc1ededee16 2023-04-27 01:56:05,551 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d75b586fc4914bbe9fd71dc1ededee16, entries=150, sequenceid=588, filesize=11.9 K 2023-04-27 01:56:05,552 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/ce8deaefcb5641d5bc0c7a72ba2b6370 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ce8deaefcb5641d5bc0c7a72ba2b6370 2023-04-27 01:56:05,554 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ce8deaefcb5641d5bc0c7a72ba2b6370, entries=150, sequenceid=588, filesize=11.9 K 2023-04-27 01:56:05,555 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bf86160e96ad9ed4e3ec62cf4c749648 in 1251ms, sequenceid=588, compaction requested=true 2023-04-27 01:56:05,555 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:05,555 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:05,555 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:05,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:05,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,557 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad53b90[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:05,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:05,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,559 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:05,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,559 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:05,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:05,560 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:05,562 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:05,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:05,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:05,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:05,563 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:05,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:05,564 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:05,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:05,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:05,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:05,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:05,566 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:05,568 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:05,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:05,568 DEBUG [Thread-2927] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:05,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,568 DEBUG [Thread-2927] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:05,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:05,568 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:05,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:05,587 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 01:56:05,587 DEBUG [Thread-2923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50150525 to 127.0.0.1:52731 2023-04-27 01:56:05,587 DEBUG [Thread-2923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:05,588 DEBUG [Thread-2919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15840296 to 127.0.0.1:52731 2023-04-27 01:56:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:05,588 DEBUG [Thread-2919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:05,588 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:05,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:05,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:05,590 DEBUG [Thread-2917] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ab56e54 to 127.0.0.1:52731 2023-04-27 01:56:05,591 DEBUG [Thread-2917] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:05,593 DEBUG [Thread-2925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x028a3a77 to 127.0.0.1:52731 2023-04-27 01:56:05,593 DEBUG [Thread-2925] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:05,668 DEBUG [Thread-2927] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:05,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:05,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21 2023-04-27 01:56:06,403 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/b2c2dcd69f8743c8808673868bf28f28 2023-04-27 01:56:06,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/9499947585b74ed58a7007b267c3e88c 2023-04-27 01:56:06,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21 2023-04-27 01:56:06,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21, entries=150, sequenceid=606, filesize=11.9 K 2023-04-27 01:56:06,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/b2c2dcd69f8743c8808673868bf28f28 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b2c2dcd69f8743c8808673868bf28f28 2023-04-27 01:56:06,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b2c2dcd69f8743c8808673868bf28f28, entries=150, sequenceid=606, filesize=11.9 K 2023-04-27 01:56:06,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/9499947585b74ed58a7007b267c3e88c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/9499947585b74ed58a7007b267c3e88c 2023-04-27 01:56:06,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/9499947585b74ed58a7007b267c3e88c, entries=150, sequenceid=606, filesize=11.9 K 2023-04-27 01:56:06,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=13.42 KB/13740 for bf86160e96ad9ed4e3ec62cf4c749648 in 1239ms, sequenceid=606, compaction requested=true 2023-04-27 01:56:06,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:06,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:06,826 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:06,826 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:06,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:06,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:06,827 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:06,827 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/A is initiating minor compaction (all files) 2023-04-27 01:56:06,827 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:06,827 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/A in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:06,827 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/B is initiating minor compaction (all files) 2023-04-27 01:56:06,827 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cb8ad60504504d6b9c7fa4ad17acb475, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5d3999597bd74922b9803102c0bb05c1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f6eae87f24f1439b9e4d51ac0f4219f1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=48.7 K 2023-04-27 01:56:06,827 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/B in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:06,827 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3ce6ae80db664cec94c21b991be1c19f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2218a3eb443747938bf5bbfdc7f757e7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d75b586fc4914bbe9fd71dc1ededee16, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b2c2dcd69f8743c8808673868bf28f28] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=48.7 K 2023-04-27 01:56:06,828 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cb8ad60504504d6b9c7fa4ad17acb475, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682560563159 2023-04-27 01:56:06,828 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3ce6ae80db664cec94c21b991be1c19f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682560563159 2023-04-27 01:56:06,828 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 5d3999597bd74922b9803102c0bb05c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682560563182 2023-04-27 01:56:06,828 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 2218a3eb443747938bf5bbfdc7f757e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682560563182 2023-04-27 01:56:06,828 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f6eae87f24f1439b9e4d51ac0f4219f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682560563828 2023-04-27 01:56:06,828 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d75b586fc4914bbe9fd71dc1ededee16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682560563828 2023-04-27 01:56:06,828 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 2ad0b67fdc6d4cf78ff5ced87cb2fb21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682560564467 2023-04-27 01:56:06,829 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b2c2dcd69f8743c8808673868bf28f28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682560564467 2023-04-27 01:56:06,835 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#A#compaction#492 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:06,836 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#B#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:07,242 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/54a38570db4b4c11bac1c78983826b71 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/54a38570db4b4c11bac1c78983826b71 2023-04-27 01:56:07,242 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/2e30bc36a1604574b01680dfc230871b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e30bc36a1604574b01680dfc230871b 2023-04-27 01:56:07,246 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/B of bf86160e96ad9ed4e3ec62cf4c749648 into 54a38570db4b4c11bac1c78983826b71(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:07,246 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/A of bf86160e96ad9ed4e3ec62cf4c749648 into 2e30bc36a1604574b01680dfc230871b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:07,246 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:07,246 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:07,246 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/B, priority=12, startTime=1682560566826; duration=0sec 2023-04-27 01:56:07,246 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/A, priority=12, startTime=1682560566826; duration=0sec 2023-04-27 01:56:07,246 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:07,246 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:07,246 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:07,247 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:07,247 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): bf86160e96ad9ed4e3ec62cf4c749648/C is initiating minor compaction (all files) 2023-04-27 01:56:07,247 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf86160e96ad9ed4e3ec62cf4c749648/C in TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:07,247 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/7a3a8a7b3846475fa8f4765dae460e28, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/aeba0c64ea014b4a9117c46a528447c7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ce8deaefcb5641d5bc0c7a72ba2b6370, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/9499947585b74ed58a7007b267c3e88c] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp, totalSize=48.7 K 2023-04-27 01:56:07,247 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7a3a8a7b3846475fa8f4765dae460e28, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1682560563159 2023-04-27 01:56:07,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting aeba0c64ea014b4a9117c46a528447c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1682560563182 2023-04-27 01:56:07,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ce8deaefcb5641d5bc0c7a72ba2b6370, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682560563828 2023-04-27 01:56:07,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9499947585b74ed58a7007b267c3e88c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1682560564467 2023-04-27 01:56:07,253 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): bf86160e96ad9ed4e3ec62cf4c749648#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:07,660 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/1ded62464b68460c8c97890eb3b63641 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1ded62464b68460c8c97890eb3b63641 2023-04-27 01:56:07,663 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf86160e96ad9ed4e3ec62cf4c749648/C of bf86160e96ad9ed4e3ec62cf4c749648 into 1ded62464b68460c8c97890eb3b63641(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:07,663 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:07,663 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648., storeName=bf86160e96ad9ed4e3ec62cf4c749648/C, priority=12, startTime=1682560566826; duration=0sec 2023-04-27 01:56:07,663 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:10,446 DEBUG [Thread-2921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3816bdf5 to 127.0.0.1:52731 2023-04-27 01:56:10,446 DEBUG [Thread-2921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:10,446 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 01:56:10,446 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-27 01:56:10,446 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-04-27 01:56:10,446 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1847 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 5541 rows 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1837 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 5511 rows 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1838 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 5514 rows 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1839 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 5517 rows 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(402): scanned 1858 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(403): verified 5568 rows 2023-04-27 01:56:10,447 INFO [Listener at localhost/42495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 01:56:10,447 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e0b6187 to 127.0.0.1:52731 2023-04-27 01:56:10,447 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:10,448 INFO [Listener at localhost/42495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 01:56:10,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 01:56:10,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:10,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 01:56:10,453 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560570453"}]},"ts":"1682560570453"} 2023-04-27 01:56:10,454 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 01:56:10,457 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 01:56:10,458 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf86160e96ad9ed4e3ec62cf4c749648, UNASSIGN}] 2023-04-27 01:56:10,459 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf86160e96ad9ed4e3ec62cf4c749648, UNASSIGN 2023-04-27 01:56:10,459 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=bf86160e96ad9ed4e3ec62cf4c749648, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:10,460 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560570459"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560570459"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560570459"}]},"ts":"1682560570459"} 2023-04-27 01:56:10,460 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:56:10,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 01:56:10,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:10,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bf86160e96ad9ed4e3ec62cf4c749648, disabling compactions & flushes 2023-04-27 01:56:10,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:10,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:10,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. after waiting 0 ms 2023-04-27 01:56:10,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:10,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bf86160e96ad9ed4e3ec62cf4c749648 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-27 01:56:10,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=A 2023-04-27 01:56:10,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:10,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=B 2023-04-27 01:56:10,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:10,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf86160e96ad9ed4e3ec62cf4c749648, store=C 2023-04-27 01:56:10,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 01:56:11,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/4146739d48fe41438653c071f3e3c11d 2023-04-27 01:56:11,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 01:56:11,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/da0d90418cae4ae292305ed943ee8e7f 2023-04-27 01:56:11,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 01:56:11,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/23d57cab97ed4bcbad4e3360670463e8 2023-04-27 01:56:11,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/A/4146739d48fe41438653c071f3e3c11d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4146739d48fe41438653c071f3e3c11d 2023-04-27 01:56:11,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4146739d48fe41438653c071f3e3c11d, entries=100, sequenceid=615, filesize=9.5 K 2023-04-27 01:56:11,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/B/da0d90418cae4ae292305ed943ee8e7f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/da0d90418cae4ae292305ed943ee8e7f 2023-04-27 01:56:11,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/da0d90418cae4ae292305ed943ee8e7f, entries=100, sequenceid=615, filesize=9.5 K 2023-04-27 01:56:11,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/.tmp/C/23d57cab97ed4bcbad4e3360670463e8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/23d57cab97ed4bcbad4e3360670463e8 2023-04-27 01:56:11,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/23d57cab97ed4bcbad4e3360670463e8, entries=100, sequenceid=615, filesize=9.5 K 2023-04-27 01:56:11,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for bf86160e96ad9ed4e3ec62cf4c749648 in 1236ms, sequenceid=615, compaction requested=false 2023-04-27 01:56:11,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/7314b61879f84f3fb8585415310d26d8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f5ab03601ed548d1b0a33e8abf6c84b2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/01450f4240db45cb973d24ee8e68120f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0b4d6149a5254de699b6ef1da56dd511, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/07984863e3564e519ceb1357c7baa7fc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a846e5b16a24469699ca9fa7088102e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f980016be92249ca84472dbfbe6b203e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/73ff59072e1844fb944f106f54f0d640, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/71a9bae941be4b4cb28f2b0b3d91f6c8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/56bcd2306e5345a5a76dbdd0b10feeb0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8247508290294de3b9af11817a8d45c7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/636a1bb211654aa4a31d3fc325656d7d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c2e548392c224e7293d461f6a55971d4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c8a824b5979544b5941df5188cf2bc5b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8a67789e77a048d5b43ad11d2ed4d91c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/097dfc3f75f54840ae734118f1be1cf0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e7860a417854486a4a80bfe0bb67d57, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4c78374d4d8644018e6c7a48ca70d581, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/ec27df05cb7c4de293d4f8682c4b47f0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a557a1ffee3c4bba8b35136c13a09f48, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/6c32a06b23164467b63fd745c7aaa5ae, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4da29c656aa94b268253fdb158a22f1b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/fbca135e04e34fceafcb6cc3cb3e038d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/70b708a5fdce48eea7d6894b16d24c7b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/69daad13ed3f4254aa1e362967c6f96a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9a52bd5279ac4837850dd8725b4cc160, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3df98383918747cb921b04c6ecad09f8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0dfa10ae61844534977c297c8fc117db, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/1f41b66a965945d88b22abb484d97ec6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5787e7bd246e44b0ba0a38d57ff49a64, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9202a3f8285941cda2da5ab6f0907a57, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/90f5985cb1c34f3ca6b2886c0405e75d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3930737e22164f22945d4a597bd92940, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cb8ad60504504d6b9c7fa4ad17acb475, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5d3999597bd74922b9803102c0bb05c1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f6eae87f24f1439b9e4d51ac0f4219f1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21] to archive 2023-04-27 01:56:11,850 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:56:11,851 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/7314b61879f84f3fb8585415310d26d8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/7314b61879f84f3fb8585415310d26d8 2023-04-27 01:56:11,852 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f5ab03601ed548d1b0a33e8abf6c84b2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f5ab03601ed548d1b0a33e8abf6c84b2 2023-04-27 01:56:11,853 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/01450f4240db45cb973d24ee8e68120f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/01450f4240db45cb973d24ee8e68120f 2023-04-27 01:56:11,854 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0b4d6149a5254de699b6ef1da56dd511 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0b4d6149a5254de699b6ef1da56dd511 2023-04-27 01:56:11,855 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/07984863e3564e519ceb1357c7baa7fc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/07984863e3564e519ceb1357c7baa7fc 2023-04-27 01:56:11,856 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a846e5b16a24469699ca9fa7088102e5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a846e5b16a24469699ca9fa7088102e5 2023-04-27 01:56:11,857 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f980016be92249ca84472dbfbe6b203e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f980016be92249ca84472dbfbe6b203e 2023-04-27 01:56:11,858 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/73ff59072e1844fb944f106f54f0d640 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/73ff59072e1844fb944f106f54f0d640 2023-04-27 01:56:11,859 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/71a9bae941be4b4cb28f2b0b3d91f6c8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/71a9bae941be4b4cb28f2b0b3d91f6c8 2023-04-27 01:56:11,860 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/56bcd2306e5345a5a76dbdd0b10feeb0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/56bcd2306e5345a5a76dbdd0b10feeb0 2023-04-27 01:56:11,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8247508290294de3b9af11817a8d45c7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8247508290294de3b9af11817a8d45c7 2023-04-27 01:56:11,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/636a1bb211654aa4a31d3fc325656d7d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/636a1bb211654aa4a31d3fc325656d7d 2023-04-27 01:56:11,862 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c2e548392c224e7293d461f6a55971d4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c2e548392c224e7293d461f6a55971d4 2023-04-27 01:56:11,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c8a824b5979544b5941df5188cf2bc5b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/c8a824b5979544b5941df5188cf2bc5b 2023-04-27 01:56:11,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8a67789e77a048d5b43ad11d2ed4d91c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/8a67789e77a048d5b43ad11d2ed4d91c 2023-04-27 01:56:11,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/097dfc3f75f54840ae734118f1be1cf0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/097dfc3f75f54840ae734118f1be1cf0 2023-04-27 01:56:11,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e7860a417854486a4a80bfe0bb67d57 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e7860a417854486a4a80bfe0bb67d57 2023-04-27 01:56:11,866 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4c78374d4d8644018e6c7a48ca70d581 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4c78374d4d8644018e6c7a48ca70d581 2023-04-27 01:56:11,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/ec27df05cb7c4de293d4f8682c4b47f0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/ec27df05cb7c4de293d4f8682c4b47f0 2023-04-27 01:56:11,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a557a1ffee3c4bba8b35136c13a09f48 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/a557a1ffee3c4bba8b35136c13a09f48 2023-04-27 01:56:11,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/6c32a06b23164467b63fd745c7aaa5ae to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/6c32a06b23164467b63fd745c7aaa5ae 2023-04-27 01:56:11,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4da29c656aa94b268253fdb158a22f1b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4da29c656aa94b268253fdb158a22f1b 2023-04-27 01:56:11,870 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/fbca135e04e34fceafcb6cc3cb3e038d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/fbca135e04e34fceafcb6cc3cb3e038d 2023-04-27 01:56:11,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/70b708a5fdce48eea7d6894b16d24c7b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/70b708a5fdce48eea7d6894b16d24c7b 2023-04-27 01:56:11,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/69daad13ed3f4254aa1e362967c6f96a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/69daad13ed3f4254aa1e362967c6f96a 2023-04-27 01:56:11,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9a52bd5279ac4837850dd8725b4cc160 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9a52bd5279ac4837850dd8725b4cc160 2023-04-27 01:56:11,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3df98383918747cb921b04c6ecad09f8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3df98383918747cb921b04c6ecad09f8 2023-04-27 01:56:11,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cdd4d66116bc41c3a1ef73ee1cbbfbc3 2023-04-27 01:56:11,875 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0dfa10ae61844534977c297c8fc117db to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/0dfa10ae61844534977c297c8fc117db 2023-04-27 01:56:11,876 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/1f41b66a965945d88b22abb484d97ec6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/1f41b66a965945d88b22abb484d97ec6 2023-04-27 01:56:11,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5787e7bd246e44b0ba0a38d57ff49a64 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5787e7bd246e44b0ba0a38d57ff49a64 2023-04-27 01:56:11,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9202a3f8285941cda2da5ab6f0907a57 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/9202a3f8285941cda2da5ab6f0907a57 2023-04-27 01:56:11,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/90f5985cb1c34f3ca6b2886c0405e75d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/90f5985cb1c34f3ca6b2886c0405e75d 2023-04-27 01:56:11,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3930737e22164f22945d4a597bd92940 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/3930737e22164f22945d4a597bd92940 2023-04-27 01:56:11,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cb8ad60504504d6b9c7fa4ad17acb475 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/cb8ad60504504d6b9c7fa4ad17acb475 2023-04-27 01:56:11,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5d3999597bd74922b9803102c0bb05c1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/5d3999597bd74922b9803102c0bb05c1 2023-04-27 01:56:11,881 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f6eae87f24f1439b9e4d51ac0f4219f1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/f6eae87f24f1439b9e4d51ac0f4219f1 2023-04-27 01:56:11,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2ad0b67fdc6d4cf78ff5ced87cb2fb21 2023-04-27 01:56:11,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ca5dcc1a6a2d42c4ad360d551d4bf31f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/bf7f89686359479086592b02f5a47df3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f081a4fc570245fe9fc3b5e3e24bff4b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6de73f0e987e4fdb830346344e4f2a91, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/feb65e14f6d348f28dced02a4c2c1f53, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/77b3b54e763243ad99374ae64c45bacb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9dbf656c98e4958a9e876336c8f12cc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/7f046340181b4b978cdf8c73aa716a67, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/59611f5ef72a4f009f813afb6be5c299, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e31796ad8e4c49e18f86598315ec6b80, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6efbe9b006dc4394b15244252f2f9864, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/011f4dcf3d344d578886197f0115781e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/975d3ed8c1d04aefba73cb472d937fba, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/16ceeb0a426b4875b3d96c92c00c7b37, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/85fc2ba2ff1145d1b2736c028d8376df, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e3d585e12c11413abf2c2804f89ceb47, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/c3e2345825624f4b9258b38a10d4f8fb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2538ad199cbb4e1aa2f0b336be278b7f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2c1a66729dff4217bb9ba1a5f2b16f11, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b6b24a59ef05413093fab6ee3ed276d5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b805b000daf84dc9be3de41f93808766, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e46c6538bdaa4236be5f9ca30bca25ff, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f7a528e3de20457aa069bce39ac69ff2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/73f9085d84274e44aa22f9faaf1b7283, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/66529cdb6b594594a1410cd6fd0d0335, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/51aa18297c184d8088545df84251abaa, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/5e68592032564897866f85d4ae0cc82d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/86f63ed9614f4dc9843cdc7b43dd7d8f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/fa2979b1189c4f2d86257f098aeecadb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9d9b41647fb4c3dbd3d8c42ee31888d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ae8a1f3b307744c79dd1d60eb205c68c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3b3eb2315dfd4bcbb7fb471992f0e01d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/12d327e3583a4796beda701170370351, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3ce6ae80db664cec94c21b991be1c19f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e0c07f56149545f29287328bbad31c82, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2218a3eb443747938bf5bbfdc7f757e7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d75b586fc4914bbe9fd71dc1ededee16, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b2c2dcd69f8743c8808673868bf28f28] to archive 2023-04-27 01:56:11,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:56:11,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ca5dcc1a6a2d42c4ad360d551d4bf31f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ca5dcc1a6a2d42c4ad360d551d4bf31f 2023-04-27 01:56:11,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/bf7f89686359479086592b02f5a47df3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/bf7f89686359479086592b02f5a47df3 2023-04-27 01:56:11,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f081a4fc570245fe9fc3b5e3e24bff4b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f081a4fc570245fe9fc3b5e3e24bff4b 2023-04-27 01:56:11,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6de73f0e987e4fdb830346344e4f2a91 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6de73f0e987e4fdb830346344e4f2a91 2023-04-27 01:56:11,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/feb65e14f6d348f28dced02a4c2c1f53 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/feb65e14f6d348f28dced02a4c2c1f53 2023-04-27 01:56:11,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/77b3b54e763243ad99374ae64c45bacb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/77b3b54e763243ad99374ae64c45bacb 2023-04-27 01:56:11,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9dbf656c98e4958a9e876336c8f12cc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9dbf656c98e4958a9e876336c8f12cc 2023-04-27 01:56:11,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/7f046340181b4b978cdf8c73aa716a67 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/7f046340181b4b978cdf8c73aa716a67 2023-04-27 01:56:11,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/59611f5ef72a4f009f813afb6be5c299 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/59611f5ef72a4f009f813afb6be5c299 2023-04-27 01:56:11,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e31796ad8e4c49e18f86598315ec6b80 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e31796ad8e4c49e18f86598315ec6b80 2023-04-27 01:56:11,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6efbe9b006dc4394b15244252f2f9864 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/6efbe9b006dc4394b15244252f2f9864 2023-04-27 01:56:11,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/011f4dcf3d344d578886197f0115781e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/011f4dcf3d344d578886197f0115781e 2023-04-27 01:56:11,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/975d3ed8c1d04aefba73cb472d937fba to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/975d3ed8c1d04aefba73cb472d937fba 2023-04-27 01:56:11,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/16ceeb0a426b4875b3d96c92c00c7b37 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/16ceeb0a426b4875b3d96c92c00c7b37 2023-04-27 01:56:11,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/85fc2ba2ff1145d1b2736c028d8376df to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/85fc2ba2ff1145d1b2736c028d8376df 2023-04-27 01:56:11,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e3d585e12c11413abf2c2804f89ceb47 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e3d585e12c11413abf2c2804f89ceb47 2023-04-27 01:56:11,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/c3e2345825624f4b9258b38a10d4f8fb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/c3e2345825624f4b9258b38a10d4f8fb 2023-04-27 01:56:11,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2538ad199cbb4e1aa2f0b336be278b7f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2538ad199cbb4e1aa2f0b336be278b7f 2023-04-27 01:56:11,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2c1a66729dff4217bb9ba1a5f2b16f11 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2c1a66729dff4217bb9ba1a5f2b16f11 2023-04-27 01:56:11,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b6b24a59ef05413093fab6ee3ed276d5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b6b24a59ef05413093fab6ee3ed276d5 2023-04-27 01:56:11,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b805b000daf84dc9be3de41f93808766 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b805b000daf84dc9be3de41f93808766 2023-04-27 01:56:11,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e46c6538bdaa4236be5f9ca30bca25ff to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e46c6538bdaa4236be5f9ca30bca25ff 2023-04-27 01:56:11,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f7a528e3de20457aa069bce39ac69ff2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/f7a528e3de20457aa069bce39ac69ff2 2023-04-27 01:56:11,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/73f9085d84274e44aa22f9faaf1b7283 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/73f9085d84274e44aa22f9faaf1b7283 2023-04-27 01:56:11,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/66529cdb6b594594a1410cd6fd0d0335 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/66529cdb6b594594a1410cd6fd0d0335 2023-04-27 01:56:11,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/51aa18297c184d8088545df84251abaa to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/51aa18297c184d8088545df84251abaa 2023-04-27 01:56:11,905 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/5e68592032564897866f85d4ae0cc82d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/5e68592032564897866f85d4ae0cc82d 2023-04-27 01:56:11,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/86f63ed9614f4dc9843cdc7b43dd7d8f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/86f63ed9614f4dc9843cdc7b43dd7d8f 2023-04-27 01:56:11,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/fa2979b1189c4f2d86257f098aeecadb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/fa2979b1189c4f2d86257f098aeecadb 2023-04-27 01:56:11,907 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9d9b41647fb4c3dbd3d8c42ee31888d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d9d9b41647fb4c3dbd3d8c42ee31888d 2023-04-27 01:56:11,908 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ae8a1f3b307744c79dd1d60eb205c68c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/ae8a1f3b307744c79dd1d60eb205c68c 2023-04-27 01:56:11,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3b3eb2315dfd4bcbb7fb471992f0e01d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3b3eb2315dfd4bcbb7fb471992f0e01d 2023-04-27 01:56:11,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/12d327e3583a4796beda701170370351 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/12d327e3583a4796beda701170370351 2023-04-27 01:56:11,910 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3ce6ae80db664cec94c21b991be1c19f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/3ce6ae80db664cec94c21b991be1c19f 2023-04-27 01:56:11,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e0c07f56149545f29287328bbad31c82 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/e0c07f56149545f29287328bbad31c82 2023-04-27 01:56:11,912 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2218a3eb443747938bf5bbfdc7f757e7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/2218a3eb443747938bf5bbfdc7f757e7 2023-04-27 01:56:11,913 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d75b586fc4914bbe9fd71dc1ededee16 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/d75b586fc4914bbe9fd71dc1ededee16 2023-04-27 01:56:11,913 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b2c2dcd69f8743c8808673868bf28f28 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/b2c2dcd69f8743c8808673868bf28f28 2023-04-27 01:56:11,916 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/847858b7cbde4fa583e551c49018c87b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/deb0a18c44ab4298bb1f95fb51fdc946, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/107b502d3a644a3d897fdf4d0b8c2cf7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3fb7434e69624c96a89caf1ff3892fcd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/d91e599372ac41238d97dee0e58e7cd9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/eb3b273532294a8a87eafe2ecbab3dea, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/956d0ad87c16490db6939b2ee05982b0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/c4b87ab2d2d34ab2964dbaa620fe2ff1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2ed7f3c0664b46348e716cb4178a086d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4b42284b23bc47bcb969ede77755edeb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fdcd279820264838ac611f34a37f2f73, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ff73f3a4c92d4f89825c3a41d63ea5a4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/6691f53ed89043829fb0e82980ea806b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1d73eaba34d541f490537ca6facd822a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b53fb8bca0ab4e6982a49e952fe9382f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/af5268f550654350bbafde45b35094fe, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f67932f1e6e64e4ab006d3ca4eacb52b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/84090761666b4393a8bab35774ca2fc2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/a8365797acff482abe1fde143a2f4ddc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b5064c7dbab546bab75b76fd8277b78a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/8528d83503104af38879579092cce630, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/95c588d5408b45cdbc59e68d94dcfde1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4f93237789104965a5ce4944671d4735, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fe7e5c4feff743f380107802bae9c4d6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3e04dda007084f4d85308b800f9dd00e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/90242be229104372821add8272101ca9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b47c0889a92d4217a2b03810f78f2d1d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b495461c85df498f8cb5b84cf03b24f5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/dcc765d22cfd4c2f8f42b409aa823dd5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/32014020d82b4d51a670d34d64760583, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/51bf0cf1ed7d474cb77db50de0ff5a5b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/bbaa1daf05944efa8ef08e9ff748a50b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f6d10b5fc38d4cfdab78fbe711976cff, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/7a3a8a7b3846475fa8f4765dae460e28, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2c92de635ea84e6fbb190a7be07396d9, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/aeba0c64ea014b4a9117c46a528447c7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ce8deaefcb5641d5bc0c7a72ba2b6370, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/9499947585b74ed58a7007b267c3e88c] to archive 2023-04-27 01:56:11,917 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:56:11,918 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/847858b7cbde4fa583e551c49018c87b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/847858b7cbde4fa583e551c49018c87b 2023-04-27 01:56:11,919 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/deb0a18c44ab4298bb1f95fb51fdc946 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/deb0a18c44ab4298bb1f95fb51fdc946 2023-04-27 01:56:11,920 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/107b502d3a644a3d897fdf4d0b8c2cf7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/107b502d3a644a3d897fdf4d0b8c2cf7 2023-04-27 01:56:11,921 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3fb7434e69624c96a89caf1ff3892fcd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3fb7434e69624c96a89caf1ff3892fcd 2023-04-27 01:56:11,922 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/d91e599372ac41238d97dee0e58e7cd9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/d91e599372ac41238d97dee0e58e7cd9 2023-04-27 01:56:11,923 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/eb3b273532294a8a87eafe2ecbab3dea to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/eb3b273532294a8a87eafe2ecbab3dea 2023-04-27 01:56:11,923 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/956d0ad87c16490db6939b2ee05982b0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/956d0ad87c16490db6939b2ee05982b0 2023-04-27 01:56:11,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/c4b87ab2d2d34ab2964dbaa620fe2ff1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/c4b87ab2d2d34ab2964dbaa620fe2ff1 2023-04-27 01:56:11,925 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2ed7f3c0664b46348e716cb4178a086d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2ed7f3c0664b46348e716cb4178a086d 2023-04-27 01:56:11,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4b42284b23bc47bcb969ede77755edeb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4b42284b23bc47bcb969ede77755edeb 2023-04-27 01:56:11,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fdcd279820264838ac611f34a37f2f73 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fdcd279820264838ac611f34a37f2f73 2023-04-27 01:56:11,929 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ff73f3a4c92d4f89825c3a41d63ea5a4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ff73f3a4c92d4f89825c3a41d63ea5a4 2023-04-27 01:56:11,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/6691f53ed89043829fb0e82980ea806b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/6691f53ed89043829fb0e82980ea806b 2023-04-27 01:56:11,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1d73eaba34d541f490537ca6facd822a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1d73eaba34d541f490537ca6facd822a 2023-04-27 01:56:11,931 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b53fb8bca0ab4e6982a49e952fe9382f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b53fb8bca0ab4e6982a49e952fe9382f 2023-04-27 01:56:11,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/af5268f550654350bbafde45b35094fe to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/af5268f550654350bbafde45b35094fe 2023-04-27 01:56:11,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f67932f1e6e64e4ab006d3ca4eacb52b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f67932f1e6e64e4ab006d3ca4eacb52b 2023-04-27 01:56:11,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/84090761666b4393a8bab35774ca2fc2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/84090761666b4393a8bab35774ca2fc2 2023-04-27 01:56:11,935 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/a8365797acff482abe1fde143a2f4ddc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/a8365797acff482abe1fde143a2f4ddc 2023-04-27 01:56:11,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b5064c7dbab546bab75b76fd8277b78a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b5064c7dbab546bab75b76fd8277b78a 2023-04-27 01:56:11,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/8528d83503104af38879579092cce630 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/8528d83503104af38879579092cce630 2023-04-27 01:56:11,937 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/95c588d5408b45cdbc59e68d94dcfde1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/95c588d5408b45cdbc59e68d94dcfde1 2023-04-27 01:56:11,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4f93237789104965a5ce4944671d4735 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/4f93237789104965a5ce4944671d4735 2023-04-27 01:56:11,939 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fe7e5c4feff743f380107802bae9c4d6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/fe7e5c4feff743f380107802bae9c4d6 2023-04-27 01:56:11,939 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3e04dda007084f4d85308b800f9dd00e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/3e04dda007084f4d85308b800f9dd00e 2023-04-27 01:56:11,940 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/90242be229104372821add8272101ca9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/90242be229104372821add8272101ca9 2023-04-27 01:56:11,941 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b47c0889a92d4217a2b03810f78f2d1d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b47c0889a92d4217a2b03810f78f2d1d 2023-04-27 01:56:11,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b495461c85df498f8cb5b84cf03b24f5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/b495461c85df498f8cb5b84cf03b24f5 2023-04-27 01:56:11,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/dcc765d22cfd4c2f8f42b409aa823dd5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/dcc765d22cfd4c2f8f42b409aa823dd5 2023-04-27 01:56:11,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/32014020d82b4d51a670d34d64760583 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/32014020d82b4d51a670d34d64760583 2023-04-27 01:56:11,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/51bf0cf1ed7d474cb77db50de0ff5a5b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/51bf0cf1ed7d474cb77db50de0ff5a5b 2023-04-27 01:56:11,945 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/bbaa1daf05944efa8ef08e9ff748a50b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/bbaa1daf05944efa8ef08e9ff748a50b 2023-04-27 01:56:11,945 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f6d10b5fc38d4cfdab78fbe711976cff to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/f6d10b5fc38d4cfdab78fbe711976cff 2023-04-27 01:56:11,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/7a3a8a7b3846475fa8f4765dae460e28 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/7a3a8a7b3846475fa8f4765dae460e28 2023-04-27 01:56:11,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2c92de635ea84e6fbb190a7be07396d9 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/2c92de635ea84e6fbb190a7be07396d9 2023-04-27 01:56:11,948 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/aeba0c64ea014b4a9117c46a528447c7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/aeba0c64ea014b4a9117c46a528447c7 2023-04-27 01:56:11,949 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ce8deaefcb5641d5bc0c7a72ba2b6370 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/ce8deaefcb5641d5bc0c7a72ba2b6370 2023-04-27 01:56:11,950 DEBUG [StoreCloser-TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/9499947585b74ed58a7007b267c3e88c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/9499947585b74ed58a7007b267c3e88c 2023-04-27 01:56:11,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/recovered.edits/618.seqid, newMaxSeqId=618, maxSeqId=1 2023-04-27 01:56:11,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648. 2023-04-27 01:56:11,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bf86160e96ad9ed4e3ec62cf4c749648: 2023-04-27 01:56:11,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:11,957 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=bf86160e96ad9ed4e3ec62cf4c749648, regionState=CLOSED 2023-04-27 01:56:11,957 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560571957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560571957"}]},"ts":"1682560571957"} 2023-04-27 01:56:11,959 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-27 01:56:11,959 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure bf86160e96ad9ed4e3ec62cf4c749648, server=jenkins-hbase4.apache.org,38085,1682560417985 in 1.4980 sec 2023-04-27 01:56:11,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-27 01:56:11,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf86160e96ad9ed4e3ec62cf4c749648, UNASSIGN in 1.5010 sec 2023-04-27 01:56:11,960 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560571960"}]},"ts":"1682560571960"} 2023-04-27 01:56:11,961 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 01:56:11,962 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 01:56:11,963 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5140 sec 2023-04-27 01:56:12,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 01:56:12,558 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-27 01:56:12,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 01:56:12,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:12,560 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:12,561 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:12,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-27 01:56:12,564 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:12,566 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/recovered.edits] 2023-04-27 01:56:12,568 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e30bc36a1604574b01680dfc230871b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/2e30bc36a1604574b01680dfc230871b 2023-04-27 01:56:12,569 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4146739d48fe41438653c071f3e3c11d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/A/4146739d48fe41438653c071f3e3c11d 2023-04-27 01:56:12,570 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/54a38570db4b4c11bac1c78983826b71 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/54a38570db4b4c11bac1c78983826b71 2023-04-27 01:56:12,571 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/da0d90418cae4ae292305ed943ee8e7f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/B/da0d90418cae4ae292305ed943ee8e7f 2023-04-27 01:56:12,573 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1ded62464b68460c8c97890eb3b63641 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/1ded62464b68460c8c97890eb3b63641 2023-04-27 01:56:12,574 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/23d57cab97ed4bcbad4e3360670463e8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/C/23d57cab97ed4bcbad4e3360670463e8 2023-04-27 01:56:12,576 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/recovered.edits/618.seqid to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648/recovered.edits/618.seqid 2023-04-27 01:56:12,577 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/bf86160e96ad9ed4e3ec62cf4c749648 2023-04-27 01:56:12,577 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:56:12,578 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:12,580 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 01:56:12,581 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 01:56:12,582 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:12,582 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 01:56:12,582 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682560572582"}]},"ts":"9223372036854775807"} 2023-04-27 01:56:12,583 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 01:56:12,583 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bf86160e96ad9ed4e3ec62cf4c749648, NAME => 'TestAcidGuarantees,,1682560542579.bf86160e96ad9ed4e3ec62cf4c749648.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 01:56:12,583 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 01:56:12,583 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682560572583"}]},"ts":"9223372036854775807"} 2023-04-27 01:56:12,584 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 01:56:12,585 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:12,586 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 28 msec 2023-04-27 01:56:12,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-27 01:56:12,662 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-27 01:56:12,673 INFO [Listener at localhost/42495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 238), OpenFileDescriptor=558 (was 559), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=394 (was 386) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12339 (was 12712) 2023-04-27 01:56:12,685 INFO [Listener at localhost/42495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=558, MaxFileDescriptor=60000, SystemLoadAverage=394, ProcessCount=170, AvailableMemoryMB=12339 2023-04-27 01:56:12,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:56:12,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:56:12,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:12,689 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 01:56:12,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-27 01:56:12,690 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 01:56:12,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 01:56:12,691 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:12,691 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 empty. 2023-04-27 01:56:12,692 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:12,692 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:56:12,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 01:56:12,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 01:56:13,099 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:56:13,100 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ec4ba6d393d7d8e622c952e9d8a3e627, NAME => 'TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp 2023-04-27 01:56:13,105 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:56:13,105 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ec4ba6d393d7d8e622c952e9d8a3e627, disabling compactions & flushes 2023-04-27 01:56:13,105 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,105 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,105 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. after waiting 0 ms 2023-04-27 01:56:13,105 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,105 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,105 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:13,107 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 01:56:13,107 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560573107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560573107"}]},"ts":"1682560573107"} 2023-04-27 01:56:13,108 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 01:56:13,109 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 01:56:13,109 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560573109"}]},"ts":"1682560573109"} 2023-04-27 01:56:13,110 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 01:56:13,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, ASSIGN}] 2023-04-27 01:56:13,115 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, ASSIGN 2023-04-27 01:56:13,115 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=false 2023-04-27 01:56:13,266 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:13,267 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560573266"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560573266"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560573266"}]},"ts":"1682560573266"} 2023-04-27 01:56:13,268 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:56:13,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 01:56:13,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ec4ba6d393d7d8e622c952e9d8a3e627, NAME => 'TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:56:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:56:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,425 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,426 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A 2023-04-27 01:56:13,426 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A 2023-04-27 01:56:13,426 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:56:13,427 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec4ba6d393d7d8e622c952e9d8a3e627 columnFamilyName A 2023-04-27 01:56:13,427 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(310): Store=ec4ba6d393d7d8e622c952e9d8a3e627/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:56:13,427 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,428 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B 2023-04-27 01:56:13,428 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B 2023-04-27 01:56:13,428 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:56:13,428 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec4ba6d393d7d8e622c952e9d8a3e627 columnFamilyName B 2023-04-27 01:56:13,429 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(310): Store=ec4ba6d393d7d8e622c952e9d8a3e627/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:56:13,429 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,430 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C 2023-04-27 01:56:13,430 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C 2023-04-27 01:56:13,430 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:56:13,430 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec4ba6d393d7d8e622c952e9d8a3e627 columnFamilyName C 2023-04-27 01:56:13,431 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(310): Store=ec4ba6d393d7d8e622c952e9d8a3e627/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:56:13,431 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:56:13,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:13,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 01:56:13,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ec4ba6d393d7d8e622c952e9d8a3e627; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69132299, jitterRate=0.030151531100273132}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:56:13,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:13,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., pid=56, masterSystemTime=1682560573419 2023-04-27 01:56:13,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:13,439 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:13,439 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560573439"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560573439"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560573439"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560573439"}]},"ts":"1682560573439"} 2023-04-27 01:56:13,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-27 01:56:13,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 in 172 msec 2023-04-27 01:56:13,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-27 01:56:13,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, ASSIGN in 329 msec 2023-04-27 01:56:13,443 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 01:56:13,443 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560573443"}]},"ts":"1682560573443"} 2023-04-27 01:56:13,444 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 01:56:13,446 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 01:56:13,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 759 msec 2023-04-27 01:56:13,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 01:56:13,794 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-27 01:56:13,795 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x041dc777 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:13,799 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c0766f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:13,800 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:13,801 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43494, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:13,802 DEBUG [Listener at localhost/42495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 01:56:13,804 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 01:56:13,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 01:56:13,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 01:56:13,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:14,222 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 01:56:14,223 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 01:56:14,223 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 01:56:14,225 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-27 01:56:14,227 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, REOPEN/MOVE}] 2023-04-27 01:56:14,228 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, REOPEN/MOVE 2023-04-27 01:56:14,228 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,228 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560574228"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560574228"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560574228"}]},"ts":"1682560574228"} 2023-04-27 01:56:14,229 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:56:14,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ec4ba6d393d7d8e622c952e9d8a3e627, disabling compactions & flushes 2023-04-27 01:56:14,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. after waiting 0 ms 2023-04-27 01:56:14,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-27 01:56:14,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:14,386 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ec4ba6d393d7d8e622c952e9d8a3e627 to self. 2023-04-27 01:56:14,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,387 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=CLOSED 2023-04-27 01:56:14,387 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560574387"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560574387"}]},"ts":"1682560574387"} 2023-04-27 01:56:14,389 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-27 01:56:14,389 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 in 159 msec 2023-04-27 01:56:14,390 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38085,1682560417985; forceNewPlan=false, retain=true 2023-04-27 01:56:14,540 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,540 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560574540"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560574540"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560574540"}]},"ts":"1682560574540"} 2023-04-27 01:56:14,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:56:14,696 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ec4ba6d393d7d8e622c952e9d8a3e627, NAME => 'TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.', STARTKEY => '', ENDKEY => ''} 2023-04-27 01:56:14,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 01:56:14,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,697 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,698 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A 2023-04-27 01:56:14,698 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A 2023-04-27 01:56:14,698 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:56:14,698 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec4ba6d393d7d8e622c952e9d8a3e627 columnFamilyName A 2023-04-27 01:56:14,700 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(310): Store=ec4ba6d393d7d8e622c952e9d8a3e627/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:56:14,700 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,700 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B 2023-04-27 01:56:14,701 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B 2023-04-27 01:56:14,701 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:56:14,701 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec4ba6d393d7d8e622c952e9d8a3e627 columnFamilyName B 2023-04-27 01:56:14,701 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(310): Store=ec4ba6d393d7d8e622c952e9d8a3e627/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:56:14,701 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,702 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C 2023-04-27 01:56:14,702 DEBUG [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C 2023-04-27 01:56:14,702 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 01:56:14,702 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec4ba6d393d7d8e622c952e9d8a3e627 columnFamilyName C 2023-04-27 01:56:14,702 INFO [StoreOpener-ec4ba6d393d7d8e622c952e9d8a3e627-1] regionserver.HStore(310): Store=ec4ba6d393d7d8e622c952e9d8a3e627/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 01:56:14,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 01:56:14,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ec4ba6d393d7d8e622c952e9d8a3e627; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64625756, jitterRate=-0.037001192569732666}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 01:56:14,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:14,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., pid=61, masterSystemTime=1682560574692 2023-04-27 01:56:14,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,709 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,709 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560574709"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682560574709"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682560574709"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682560574709"}]},"ts":"1682560574709"} 2023-04-27 01:56:14,711 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-27 01:56:14,711 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 in 169 msec 2023-04-27 01:56:14,713 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-27 01:56:14,713 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, REOPEN/MOVE in 484 msec 2023-04-27 01:56:14,714 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-27 01:56:14,715 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-27 01:56:14,716 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 909 msec 2023-04-27 01:56:14,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-27 01:56:14,717 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x089b6154 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,722 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6257294f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,723 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x01d207bb to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,727 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52104714, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,727 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x5bcc330a to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,732 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a320817, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,732 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x23afc0f2 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,736 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19c1e3ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,737 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x0f1adb60 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,740 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@526f2221, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,741 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x17cf2e3f to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,752 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c1a68cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,752 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x3420e54c to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,757 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a709b21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,757 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x26d7d0b5 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,765 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@156e4ba6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,765 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x601410a4 to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,769 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@785e366b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,769 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(139): Connect 0x1f8e4d5e to 127.0.0.1:52731 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 01:56:14,774 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fdfef3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 01:56:14,785 DEBUG [hconnection-0x4580dc0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,785 DEBUG [hconnection-0x2f258d32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,787 DEBUG [hconnection-0x28972dfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,787 DEBUG [hconnection-0x5375fba8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,788 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,794 DEBUG [hconnection-0x4927eb99-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,793 DEBUG [hconnection-0x545952c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,794 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,794 DEBUG [hconnection-0x4306b761-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,795 DEBUG [hconnection-0x14e0ebfa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,794 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,797 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,797 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,798 DEBUG [hconnection-0x3877f414-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,798 DEBUG [hconnection-0x75a7abd6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 01:56:14,802 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,803 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,803 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43538, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,805 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43550, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,810 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 01:56:14,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:14,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:14,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 01:56:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:14,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:14,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:14,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:14,826 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:14,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:14,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:14,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:14,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:14,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:14,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,828 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:14,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:14,832 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,832 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:14,833 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:14,833 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,837 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,838 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:14,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:14,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b79fbff[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:14,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:14,839 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560634837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,840 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560634837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,841 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:14,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560634838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,841 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:14,841 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:14,841 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:14,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560634839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560634841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,843 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:14,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,844 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:14,844 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:14,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:14,846 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:14,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:14,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:14,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:14,846 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:14,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,847 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:14,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:14,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:14,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,850 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,851 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:14,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:14,852 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:14,852 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:14,852 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:14,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:14,852 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:14,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:14,874 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427f6244a777aa44a81825a9957794477f6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f6244a777aa44a81825a9957794477f6 2023-04-27 01:56:14,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560634941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560634942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560634942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560634942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:14,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560634943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,952 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:14,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:14,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:14,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:14,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:14,956 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:14,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:14,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:14,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:14,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,957 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,957 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:14,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:14,957 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,958 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:14,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:14,959 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,959 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:14,959 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:14,959 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:14,959 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,961 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:14,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:14,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:14,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b81261[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:14,963 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:14,963 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:14,965 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:14,965 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,965 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:14,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:14,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:14,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:14,966 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:14,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:14,968 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:14,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:14,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:14,968 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:14,968 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:14,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,970 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:14,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:14,970 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:14,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,972 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:14,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:14,974 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:14,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:14,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:14,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:14,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:14,975 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:14,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:14,975 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:14,974 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:14,975 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:14,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:14,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,075 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:15,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:15,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:15,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:15,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:15,078 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:15,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:15,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:15,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,079 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:15,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:15,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:15,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:15,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:15,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:15,082 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,082 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:15,082 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:15,082 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:15,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:15,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:15,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:15,084 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,084 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:15,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ebdcbf[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:15,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:15,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,087 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,087 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:15,087 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:15,087 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:15,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:15,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,089 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:15,089 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,089 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:15,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:15,092 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:15,092 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:15,092 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:15,092 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:15,092 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:15,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,094 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,094 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,096 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,098 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:15,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:15,098 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,099 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:15,099 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:15,099 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560635143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560635145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560635147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560635148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560635148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,199 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:15,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:15,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:15,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:15,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:15,202 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:15,202 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,202 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:15,202 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:15,203 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,203 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,208 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,208 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,208 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:15,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:15,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:15,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:15,210 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:15,210 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,210 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:15,216 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:15,216 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:15,216 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,216 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:15,216 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:15,216 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,219 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:15,219 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:15,219 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,219 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:15,219 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:15,220 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,220 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,221 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,221 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:15,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a16689[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:15,223 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:15,223 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,224 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:15,224 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:15,224 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,224 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:15,225 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,225 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:15,225 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,225 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:15,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:15,228 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:15,228 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:15,228 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:15,228 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:15,228 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:15,231 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,231 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,231 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,232 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,233 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,235 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:15,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,235 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,235 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:15,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,235 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:15,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:15,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,286 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/4c1b49df54804349b654f3cf3e83122a 2023-04-27 01:56:15,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/128f8c843e8f4db38075ab05a03f2a98 2023-04-27 01:56:15,336 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:15,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:15,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:15,340 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:15,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:15,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:15,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,340 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,342 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,342 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,342 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,342 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:15,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:15,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:15,344 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,344 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:15,344 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:15,344 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:15,344 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,345 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:15,345 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:15,346 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,346 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,347 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:15,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f3c301[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:15,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:15,349 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,351 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:15,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,352 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:15,353 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:15,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:15,356 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:15,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:15,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:15,356 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:15,356 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:15,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,358 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,358 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,360 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,362 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:15,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,362 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,363 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:15,363 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:15,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:15,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560635448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560635448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560635448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560635451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560635452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,463 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:15,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:15,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:15,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:15,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:15,466 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:15,466 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:15,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:15,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,467 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,469 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,469 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,470 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:15,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:15,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:15,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:15,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:15,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:15,471 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:15,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,471 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:15,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:15,472 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:15,473 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,473 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,474 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,474 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e81ec84[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:15,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:15,475 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,477 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,477 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:15,477 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:15,477 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:15,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,477 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:15,478 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:15,481 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:15,481 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:15,481 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:15,481 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:15,481 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:15,484 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,484 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,484 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,486 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,489 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,489 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,489 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:15,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:15,489 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,489 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:15,489 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:15,489 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:15,489 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,490 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,490 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,490 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,490 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,590 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:15,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:15,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:15,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:15,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:15,593 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:15,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:15,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:15,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,593 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,594 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,595 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,595 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:15,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:15,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:15,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:15,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:15,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:15,596 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:15,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:15,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:15,597 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:15,602 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,602 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,602 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,602 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:15,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3091d3cf[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:15,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:15,604 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,605 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:15,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:15,605 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:15,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,605 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:15,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:15,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,606 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,606 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:15,609 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:15,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:15,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:15,609 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:15,609 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:15,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,610 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,611 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,613 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,616 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:15,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,616 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,616 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:15,617 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:15,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:15,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,717 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:15,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:15,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:15,720 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:15,720 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,720 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:15,720 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:15,721 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,721 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,722 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,722 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,722 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:15,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:15,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:15,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:15,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:15,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:15,724 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,724 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:15,724 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:15,724 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:15,725 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:15,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@380c7e7a[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:15,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:15,727 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,728 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:15,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:15,728 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:15,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,729 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:15,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:15,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:15,734 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:15,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:15,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:15,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:15,734 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:15,736 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:15,736 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,736 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:15,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,737 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,738 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:15,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:15,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:15,742 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,742 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:15,742 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:15,742 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:15,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:15,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/f66d01f3036a4e16b4f1e6161b0ee543 2023-04-27 01:56:15,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/4c1b49df54804349b654f3cf3e83122a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4c1b49df54804349b654f3cf3e83122a 2023-04-27 01:56:15,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4c1b49df54804349b654f3cf3e83122a, entries=250, sequenceid=18, filesize=38.7 K 2023-04-27 01:56:15,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/128f8c843e8f4db38075ab05a03f2a98 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/128f8c843e8f4db38075ab05a03f2a98 2023-04-27 01:56:15,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/128f8c843e8f4db38075ab05a03f2a98, entries=150, sequenceid=18, filesize=11.6 K 2023-04-27 01:56:15,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/f66d01f3036a4e16b4f1e6161b0ee543 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f66d01f3036a4e16b4f1e6161b0ee543 2023-04-27 01:56:15,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f66d01f3036a4e16b4f1e6161b0ee543, entries=150, sequenceid=18, filesize=11.6 K 2023-04-27 01:56:15,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ec4ba6d393d7d8e622c952e9d8a3e627 in 938ms, sequenceid=18, compaction requested=false 2023-04-27 01:56:15,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:15,842 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:15,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:15,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:15,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:15,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:15,846 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:15,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:15,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:15,847 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,847 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,848 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,848 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:15,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:15,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,849 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:15,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:15,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:15,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:15,850 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:15,850 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:15,851 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:56:15,851 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:15,851 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:15,851 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:15,851 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:15,851 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:15,851 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:15,869 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304278f2ef3164c4242c7bc68efc3cf5b19a5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278f2ef3164c4242c7bc68efc3cf5b19a5 2023-04-27 01:56:15,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:15,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:15,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560635959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560635960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560635962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560635962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:15,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:15,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560635963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560636063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560636064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560636066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560636066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560636066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560636267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560636267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560636267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560636267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560636268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,278 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/1a5fc6286e6b472193b082686f5b893e 2023-04-27 01:56:16,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560636569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560636570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560636570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560636570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:16,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560636572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,688 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/098edf3a5f6248c497f02edba3c59f46 2023-04-27 01:56:16,703 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/23c42239e84848058ff1df1eb41e4755 2023-04-27 01:56:16,707 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/1a5fc6286e6b472193b082686f5b893e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/1a5fc6286e6b472193b082686f5b893e 2023-04-27 01:56:16,710 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/1a5fc6286e6b472193b082686f5b893e, entries=150, sequenceid=36, filesize=25.1 K 2023-04-27 01:56:16,711 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/098edf3a5f6248c497f02edba3c59f46 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/098edf3a5f6248c497f02edba3c59f46 2023-04-27 01:56:16,714 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/098edf3a5f6248c497f02edba3c59f46, entries=150, sequenceid=36, filesize=11.6 K 2023-04-27 01:56:16,715 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/23c42239e84848058ff1df1eb41e4755 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/23c42239e84848058ff1df1eb41e4755 2023-04-27 01:56:16,718 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/23c42239e84848058ff1df1eb41e4755, entries=150, sequenceid=36, filesize=11.6 K 2023-04-27 01:56:16,718 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for ec4ba6d393d7d8e622c952e9d8a3e627 in 867ms, sequenceid=36, compaction requested=false 2023-04-27 01:56:16,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:16,719 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:16,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,722 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:16,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@518e49b1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:16,724 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:16,724 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,725 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:16,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:16,725 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:16,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:16,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:16,726 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:16,727 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:16,730 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:16,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:16,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:16,730 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:16,730 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:16,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:16,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:16,732 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:16,733 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:16,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:16,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:16,737 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:16,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:16,737 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:16,738 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:16,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:16,738 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:16,738 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:16,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:16,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:16,838 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:16,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:16,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:16,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:16,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:16,842 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:16,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:16,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:16,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:16,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,842 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:16,843 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:16,843 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:16,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:16,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:16,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:16,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:16,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:16,845 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:16,845 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:16,845 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:16,845 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 01:56:16,846 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:16,846 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:16,846 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:16,846 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:16,846 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:16,846 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:16,861 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042737bf67dae7864d5ba7e4b68c5bb71a1e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042737bf67dae7864d5ba7e4b68c5bb71a1e 2023-04-27 01:56:17,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:17,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:17,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560637084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560637086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560637086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560637087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560637087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560637188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560637190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560637190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560637190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560637190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,266 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/b46aa787e8f84a97ab6dfe619d62b33b 2023-04-27 01:56:17,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560637391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560637391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560637392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560637392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560637393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,478 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 01:56:17,688 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/0d356718f7e241d0ba356e2bd5a2d77f 2023-04-27 01:56:17,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560637695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560637695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560637696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560637697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:17,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560637697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,702 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/0aecb485b2a04f3f8a71ee5390f018a4 2023-04-27 01:56:17,706 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/b46aa787e8f84a97ab6dfe619d62b33b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b46aa787e8f84a97ab6dfe619d62b33b 2023-04-27 01:56:17,710 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b46aa787e8f84a97ab6dfe619d62b33b, entries=150, sequenceid=51, filesize=25.1 K 2023-04-27 01:56:17,711 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/0d356718f7e241d0ba356e2bd5a2d77f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/0d356718f7e241d0ba356e2bd5a2d77f 2023-04-27 01:56:17,717 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/0d356718f7e241d0ba356e2bd5a2d77f, entries=150, sequenceid=51, filesize=11.6 K 2023-04-27 01:56:17,717 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/0aecb485b2a04f3f8a71ee5390f018a4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0aecb485b2a04f3f8a71ee5390f018a4 2023-04-27 01:56:17,721 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0aecb485b2a04f3f8a71ee5390f018a4, entries=150, sequenceid=51, filesize=11.6 K 2023-04-27 01:56:17,721 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ec4ba6d393d7d8e622c952e9d8a3e627 in 876ms, sequenceid=51, compaction requested=true 2023-04-27 01:56:17,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:17,722 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:17,723 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:17,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,723 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:17,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:17,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ca7ac8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:17,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:17,725 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,726 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:17,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:17,726 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:17,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,726 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:17,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:17,731 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:17,731 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:17,734 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:17,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:17,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:17,734 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:17,734 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:17,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:17,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,735 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:17,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,736 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,738 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:17,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:17,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:17,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:17,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:17,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:17,740 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:17,740 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:17,741 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:17,741 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:17,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:17,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:17,741 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,841 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:17,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:17,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:17,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:17,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:17,844 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:17,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:17,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:17,844 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:17,845 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,845 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:17,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,846 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:17,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:17,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:17,846 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:17,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:17,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:17,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:17,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:17,847 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:17,848 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:17,848 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 01:56:17,848 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:17,849 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:17,849 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:17,849 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:17,849 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:17,849 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:17,863 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427da008c8993e3437d9f50a5eded463175 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427da008c8993e3437d9f50a5eded463175 2023-04-27 01:56:18,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:18,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:18,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560638206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560638207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560638209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560638209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560638210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,275 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/d91e3bbbdf5146548f4de8a635d091e4 2023-04-27 01:56:18,287 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/e11ded7c218b406a90d15d98c1aae817 2023-04-27 01:56:18,298 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/a3c674adeb21489b98afe8f5ba00b00f 2023-04-27 01:56:18,302 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/d91e3bbbdf5146548f4de8a635d091e4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d91e3bbbdf5146548f4de8a635d091e4 2023-04-27 01:56:18,305 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d91e3bbbdf5146548f4de8a635d091e4, entries=150, sequenceid=68, filesize=25.1 K 2023-04-27 01:56:18,306 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/e11ded7c218b406a90d15d98c1aae817 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11ded7c218b406a90d15d98c1aae817 2023-04-27 01:56:18,309 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11ded7c218b406a90d15d98c1aae817, entries=150, sequenceid=68, filesize=11.6 K 2023-04-27 01:56:18,310 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/a3c674adeb21489b98afe8f5ba00b00f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/a3c674adeb21489b98afe8f5ba00b00f 2023-04-27 01:56:18,314 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/a3c674adeb21489b98afe8f5ba00b00f, entries=150, sequenceid=68, filesize=11.6 K 2023-04-27 01:56:18,314 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ec4ba6d393d7d8e622c952e9d8a3e627 in 466ms, sequenceid=68, compaction requested=true 2023-04-27 01:56:18,314 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:18,314 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,314 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:18,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:18,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 01:56:18,317 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:18,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:18,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:18,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:18,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,318 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:18,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4efbd0b5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:18,319 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:18,319 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,320 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:18,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,321 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:18,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:18,327 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:18,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:18,328 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:18,328 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:18,328 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:18,329 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042707be735249da4090af0fb81ecb1e04b5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042707be735249da4090af0fb81ecb1e04b5 2023-04-27 01:56:18,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,329 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,331 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,332 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560638329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560638330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560638331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560638331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560638332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,336 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,337 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,337 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,337 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:18,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:18,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,337 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:18,337 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:18,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560638434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,437 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:18,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560638435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:18,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:18,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:18,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:18,440 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:18,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,440 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:18,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:18,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560638437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560638437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560638437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,441 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,442 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,442 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,443 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,443 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:18,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:18,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:18,444 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,444 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:18,444 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:18,444 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:18,444 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:18,445 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:18,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,446 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:18,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15df88be[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:18,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:18,448 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,448 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:18,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:18,449 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,449 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:18,449 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:18,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:18,453 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:18,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:18,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:18,453 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:18,453 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:18,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,455 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,455 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,457 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,459 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,459 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,459 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:18,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:18,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:18,459 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,459 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:18,460 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:18,459 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,460 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,560 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:18,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:18,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:18,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:18,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:18,563 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:18,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:18,563 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:18,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:18,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:18,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:18,566 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:18,569 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,569 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bf9875[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:18,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:18,570 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,572 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:18,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:18,572 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:18,573 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,573 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:18,573 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,573 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:18,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:18,575 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:18,575 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:18,575 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:18,575 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:18,575 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:18,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,581 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,582 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,583 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,586 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:18,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,586 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,586 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,586 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:18,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560638637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560638639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560638642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560638642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560638643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,687 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:18,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:18,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:18,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:18,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:18,690 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:18,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:18,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:18,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,690 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,691 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:18,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:18,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:18,693 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,694 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:18,694 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:18,694 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:18,694 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:18,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,696 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1580b850[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:18,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:18,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:18,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:18,701 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,701 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:18,701 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,701 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:18,704 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:18,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:18,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:18,704 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:18,704 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:18,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:18,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,706 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,709 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,709 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,709 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:18,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:18,709 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:18,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:18,710 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:18,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:18,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,709 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:18,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:18,710 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:18,738 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/9cf457438a054149a29e9f9719c3a46b 2023-04-27 01:56:18,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/17c15ece270d46faba40404e0393f7bd 2023-04-27 01:56:18,770 WARN [DataStreamer for file /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/b8326f1c1e5d4418af6bc730bb212cab block BP-809979935-172.31.14.131-1682560413421:blk_1073742433_1609] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.closeInternal(DataStreamer.java:844) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:840) 2023-04-27 01:56:18,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/b8326f1c1e5d4418af6bc730bb212cab 2023-04-27 01:56:18,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/9cf457438a054149a29e9f9719c3a46b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/9cf457438a054149a29e9f9719c3a46b 2023-04-27 01:56:18,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/9cf457438a054149a29e9f9719c3a46b, entries=150, sequenceid=86, filesize=25.1 K 2023-04-27 01:56:18,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/17c15ece270d46faba40404e0393f7bd as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/17c15ece270d46faba40404e0393f7bd 2023-04-27 01:56:18,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/17c15ece270d46faba40404e0393f7bd, entries=150, sequenceid=86, filesize=11.6 K 2023-04-27 01:56:18,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/b8326f1c1e5d4418af6bc730bb212cab as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/b8326f1c1e5d4418af6bc730bb212cab 2023-04-27 01:56:18,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/b8326f1c1e5d4418af6bc730bb212cab, entries=150, sequenceid=86, filesize=11.6 K 2023-04-27 01:56:18,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for ec4ba6d393d7d8e622c952e9d8a3e627 in 479ms, sequenceid=86, compaction requested=true 2023-04-27 01:56:18,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:18,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:18,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:18,796 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:18,796 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:18,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:18,799 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:18,799 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/B is initiating minor compaction (all files) 2023-04-27 01:56:18,799 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/B in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,799 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/128f8c843e8f4db38075ab05a03f2a98, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/098edf3a5f6248c497f02edba3c59f46, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/0d356718f7e241d0ba356e2bd5a2d77f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11ded7c218b406a90d15d98c1aae817, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/17c15ece270d46faba40404e0393f7bd] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=58.0 K 2023-04-27 01:56:18,800 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 142614 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:18,800 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/A is initiating minor compaction (all files) 2023-04-27 01:56:18,800 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/A in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,800 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4c1b49df54804349b654f3cf3e83122a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/1a5fc6286e6b472193b082686f5b893e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b46aa787e8f84a97ab6dfe619d62b33b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d91e3bbbdf5146548f4de8a635d091e4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/9cf457438a054149a29e9f9719c3a46b] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=139.3 K 2023-04-27 01:56:18,800 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 128f8c843e8f4db38075ab05a03f2a98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682560574817 2023-04-27 01:56:18,800 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 4c1b49df54804349b654f3cf3e83122a, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682560574817 2023-04-27 01:56:18,800 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 098edf3a5f6248c497f02edba3c59f46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682560574831 2023-04-27 01:56:18,801 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 1a5fc6286e6b472193b082686f5b893e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682560574831 2023-04-27 01:56:18,801 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b46aa787e8f84a97ab6dfe619d62b33b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682560575958 2023-04-27 01:56:18,802 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 0d356718f7e241d0ba356e2bd5a2d77f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682560575958 2023-04-27 01:56:18,802 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d91e3bbbdf5146548f4de8a635d091e4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560577083 2023-04-27 01:56:18,803 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 9cf457438a054149a29e9f9719c3a46b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682560578315 2023-04-27 01:56:18,803 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e11ded7c218b406a90d15d98c1aae817, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560577083 2023-04-27 01:56:18,803 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 17c15ece270d46faba40404e0393f7bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682560578315 2023-04-27 01:56:18,810 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:18,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:18,818 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#B#compaction#514 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:18,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:18,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:18,818 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:18,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:18,818 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:18,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:18,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:18,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,821 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:18,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:18,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:18,842 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#A#compaction#513 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:18,852 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,852 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:18,852 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 01:56:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:18,857 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/94cf8c6aa72949f5a1ee4ac98a4bcf7d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/94cf8c6aa72949f5a1ee4ac98a4bcf7d 2023-04-27 01:56:18,861 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/B of ec4ba6d393d7d8e622c952e9d8a3e627 into 94cf8c6aa72949f5a1ee4ac98a4bcf7d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:18,861 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:18,861 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/B, priority=11, startTime=1682560578796; duration=0sec 2023-04-27 01:56:18,861 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:18,861 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:18,863 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:18,863 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/C is initiating minor compaction (all files) 2023-04-27 01:56:18,863 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/C in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:18,863 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f66d01f3036a4e16b4f1e6161b0ee543, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/23c42239e84848058ff1df1eb41e4755, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0aecb485b2a04f3f8a71ee5390f018a4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/a3c674adeb21489b98afe8f5ba00b00f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/b8326f1c1e5d4418af6bc730bb212cab] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=58.0 K 2023-04-27 01:56:18,863 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f66d01f3036a4e16b4f1e6161b0ee543, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682560574817 2023-04-27 01:56:18,863 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 23c42239e84848058ff1df1eb41e4755, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682560574831 2023-04-27 01:56:18,864 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 0aecb485b2a04f3f8a71ee5390f018a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682560575958 2023-04-27 01:56:18,864 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting a3c674adeb21489b98afe8f5ba00b00f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682560577083 2023-04-27 01:56:18,865 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b8326f1c1e5d4418af6bc730bb212cab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682560578315 2023-04-27 01:56:18,882 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#C#compaction#515 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:18,913 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/56ac11ea5da24837a23ba2b1125b0825 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/56ac11ea5da24837a23ba2b1125b0825 2023-04-27 01:56:18,917 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/C of ec4ba6d393d7d8e622c952e9d8a3e627 into 56ac11ea5da24837a23ba2b1125b0825(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:18,917 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:18,917 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/C, priority=11, startTime=1682560578796; duration=0sec 2023-04-27 01:56:18,917 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:18,922 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304276db4a3f4af7e4d73af3919463bdf82e1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276db4a3f4af7e4d73af3919463bdf82e1 2023-04-27 01:56:18,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:18,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:18,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560638956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560638957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560638967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560638967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:18,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:18,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560638967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560639068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560639068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560639070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560639071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560639071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560639271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560639272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560639276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560639276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560639276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,330 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/de021e00e84c4301a10cf5a53ab74c14 2023-04-27 01:56:19,339 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/3dcbf3f4cea24145a45360dcd0c271b3 2023-04-27 01:56:19,352 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/1f863289a6bb4f45ab3ce62df57709b5 2023-04-27 01:56:19,355 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/de021e00e84c4301a10cf5a53ab74c14 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/de021e00e84c4301a10cf5a53ab74c14 2023-04-27 01:56:19,358 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/de021e00e84c4301a10cf5a53ab74c14, entries=150, sequenceid=101, filesize=25.1 K 2023-04-27 01:56:19,359 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/3dcbf3f4cea24145a45360dcd0c271b3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3dcbf3f4cea24145a45360dcd0c271b3 2023-04-27 01:56:19,362 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3dcbf3f4cea24145a45360dcd0c271b3, entries=150, sequenceid=101, filesize=11.6 K 2023-04-27 01:56:19,362 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/1f863289a6bb4f45ab3ce62df57709b5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1f863289a6bb4f45ab3ce62df57709b5 2023-04-27 01:56:19,365 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1f863289a6bb4f45ab3ce62df57709b5, entries=150, sequenceid=101, filesize=11.6 K 2023-04-27 01:56:19,366 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for ec4ba6d393d7d8e622c952e9d8a3e627 in 514ms, sequenceid=101, compaction requested=false 2023-04-27 01:56:19,366 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:19,366 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,366 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:19,367 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:19,368 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:19,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,368 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:19,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ebc2d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:19,371 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:19,371 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,372 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,372 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,372 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:19,375 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:19,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:19,377 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:19,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:19,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:19,377 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:19,377 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:19,379 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:19,380 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:19,381 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:19,382 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:19,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:19,383 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,384 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:19,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:19,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:19,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:19,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:19,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:19,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:19,384 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:19,384 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:19,384 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:19,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:19,484 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:19,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:19,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:19,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:19,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:19,490 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:19,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:19,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:19,490 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:19,491 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,491 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:19,492 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:19,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:19,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,492 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:19,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:19,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:19,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:19,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:19,493 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:19,493 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:19,493 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:19,493 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:19,493 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:19,494 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:19,494 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:19,508 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427299219d9fdb54797a4eca9d741481e43 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427299219d9fdb54797a4eca9d741481e43 2023-04-27 01:56:19,516 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=122, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/48762217f78b4ee2930397a107f09e47 2023-04-27 01:56:19,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:19,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:19,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560639583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560639585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560639586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560639586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560639587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560639688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560639689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560639690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560639690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560639691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,721 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/c3707fe8ddd543be942337739b1c7ed5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c3707fe8ddd543be942337739b1c7ed5 2023-04-27 01:56:19,725 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/A of ec4ba6d393d7d8e622c952e9d8a3e627 into c3707fe8ddd543be942337739b1c7ed5(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:19,725 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:19,725 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/A, priority=11, startTime=1682560578796; duration=0sec 2023-04-27 01:56:19,725 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:19,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560639891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560639892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560639894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560639894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:19,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560639897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:19,928 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/4fb1e27e6e7e46a791cc7a41a2783012 2023-04-27 01:56:20,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560640195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560640195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560640197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560640197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560640201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,338 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/ef21d999337641e9be103e80ef625072 2023-04-27 01:56:20,342 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/48762217f78b4ee2930397a107f09e47 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/48762217f78b4ee2930397a107f09e47 2023-04-27 01:56:20,345 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/48762217f78b4ee2930397a107f09e47, entries=150, sequenceid=122, filesize=25.1 K 2023-04-27 01:56:20,346 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/4fb1e27e6e7e46a791cc7a41a2783012 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/4fb1e27e6e7e46a791cc7a41a2783012 2023-04-27 01:56:20,349 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/4fb1e27e6e7e46a791cc7a41a2783012, entries=150, sequenceid=122, filesize=11.6 K 2023-04-27 01:56:20,350 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/ef21d999337641e9be103e80ef625072 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/ef21d999337641e9be103e80ef625072 2023-04-27 01:56:20,352 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/ef21d999337641e9be103e80ef625072, entries=150, sequenceid=122, filesize=11.6 K 2023-04-27 01:56:20,353 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ec4ba6d393d7d8e622c952e9d8a3e627 in 859ms, sequenceid=122, compaction requested=true 2023-04-27 01:56:20,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:20,353 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:20,356 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,356 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9d4e46[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:20,359 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:20,359 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,360 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,360 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:20,360 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:20,360 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:20,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,361 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:20,361 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:20,365 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:20,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:20,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:20,365 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:20,365 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:20,366 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:20,366 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,366 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:20,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:20,367 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:20,369 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:20,378 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:20,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:20,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:20,378 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:20,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:20,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:20,379 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:20,379 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:20,479 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:20,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:20,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:20,482 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:20,482 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:20,482 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:20,482 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:20,483 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,483 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:20,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,484 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:20,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:20,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,484 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:20,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:20,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:20,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:20,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:20,485 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:20,485 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:20,485 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 01:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:20,508 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427fd428b9e85664cc6921218715534c8cd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fd428b9e85664cc6921218715534c8cd 2023-04-27 01:56:20,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:20,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:20,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560640713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560640715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560640716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560640718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560640718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560640819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560640821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560640821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560640822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:20,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560640823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:20,913 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/08f78cf8d1da4653beebdc287a91efa3 2023-04-27 01:56:21,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560641021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560641023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560641024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560641025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560641025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,324 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/9b80c85f2e134048af6c5698ab716e87 2023-04-27 01:56:21,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560641326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560641327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560641330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560641330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560641331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,737 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3de56c46de744e1d99bf601d3230e050 2023-04-27 01:56:21,741 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/08f78cf8d1da4653beebdc287a91efa3 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/08f78cf8d1da4653beebdc287a91efa3 2023-04-27 01:56:21,750 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/08f78cf8d1da4653beebdc287a91efa3, entries=150, sequenceid=136, filesize=25.3 K 2023-04-27 01:56:21,750 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/9b80c85f2e134048af6c5698ab716e87 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9b80c85f2e134048af6c5698ab716e87 2023-04-27 01:56:21,757 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9b80c85f2e134048af6c5698ab716e87, entries=150, sequenceid=136, filesize=11.8 K 2023-04-27 01:56:21,758 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3de56c46de744e1d99bf601d3230e050 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3de56c46de744e1d99bf601d3230e050 2023-04-27 01:56:21,761 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3de56c46de744e1d99bf601d3230e050, entries=150, sequenceid=136, filesize=11.8 K 2023-04-27 01:56:21,762 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ec4ba6d393d7d8e622c952e9d8a3e627 in 1277ms, sequenceid=136, compaction requested=true 2023-04-27 01:56:21,762 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:21,762 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:21,763 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262b1f90[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:21,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:21,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,767 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,767 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:21,767 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,767 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:21,767 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:21,767 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:21,769 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:21,772 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:21,772 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:21,772 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:21,772 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:21,772 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:21,773 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,773 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:21,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,773 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:21,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:21,774 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,775 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:21,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:21,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:21,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:21,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:21,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:21,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,778 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,778 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,778 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,777 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:21,778 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:21,778 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:21,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:21,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:21,833 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 01:56:21,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:21,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:21,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:21,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:21,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:21,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:21,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560641842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560641843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560641846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,851 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e268024c8d1b419eac7a202131c3cbdb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e268024c8d1b419eac7a202131c3cbdb 2023-04-27 01:56:21,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560641847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560641847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,878 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:21,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:21,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:21,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:21,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:21,881 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:21,881 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:21,881 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:21,881 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:21,882 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,882 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:21,883 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:21,883 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:21,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:21,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:21,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:21,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:21,885 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:21,885 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:21,885 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:21,885 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:21,888 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,888 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:21,889 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,889 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3679613d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:21,890 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:21,890 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,891 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,892 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:21,892 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:21,892 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:21,892 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,892 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:21,893 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,893 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:21,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:21,895 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:21,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:21,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:21,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:21,896 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:21,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:21,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,897 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:21,898 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,900 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:21,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:21,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:21,903 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:21,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:21,903 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:21,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:21,904 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:21,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:21,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560641948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560641948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560641951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:21,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560641952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:21,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560641952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,004 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:22,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:22,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:22,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:22,007 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:22,007 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,007 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:22,007 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:22,008 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,008 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:22,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:22,009 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:22,010 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,010 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:22,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:22,011 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,011 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:22,011 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:22,011 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:22,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:22,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:22,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:22,013 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,013 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,013 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,013 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46063c0b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:22,015 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:22,015 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,016 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,016 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,016 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:22,016 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:22,016 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:22,016 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:22,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:22,022 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:22,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:22,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:22,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:22,022 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:22,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,024 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:22,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,024 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,025 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:22,030 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:22,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:22,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:22,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,030 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:22,030 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:22,030 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:22,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:22,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:22,130 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:22,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:22,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:22,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:22,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:22,134 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:22,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:22,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:22,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,137 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,137 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,137 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:22,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:22,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:22,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:22,139 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,139 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:22,139 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:22,139 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:22,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,141 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,141 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:22,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40efb5f3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:22,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:22,143 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,144 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:22,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:22,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:22,148 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:22,148 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:22,148 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:22,148 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:22,149 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:22,150 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,151 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,151 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:22,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,151 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,153 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560642151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:22,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,155 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:22,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,155 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,155 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:22,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560642153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,156 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:22,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:22,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:22,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560642156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560642157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560642158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,256 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:22,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:22,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:22,258 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/7469efaa34464e168babea6a03f5e1d5 2023-04-27 01:56:22,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:22,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:22,260 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:22,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:22,260 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:22,261 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,261 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,262 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:22,262 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,263 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,263 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:22,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:22,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:22,265 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,265 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:22,265 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:22,265 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:22,267 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e46e5e1[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:22,270 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:22,270 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/994be0785ebb492a95508800e1e6850f 2023-04-27 01:56:22,272 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:22,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,273 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:22,274 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:22,274 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:22,274 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:22,274 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:22,281 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:22,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:22,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:22,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:22,281 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:22,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,282 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:22,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:22,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:22,283 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/bc777593ad314f8885e16e16b0686b08 2023-04-27 01:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,285 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:22,288 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:22,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:22,289 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:22,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,289 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:22,289 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:22,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/7469efaa34464e168babea6a03f5e1d5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7469efaa34464e168babea6a03f5e1d5 2023-04-27 01:56:22,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:22,289 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:22,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7469efaa34464e168babea6a03f5e1d5, entries=250, sequenceid=157, filesize=38.9 K 2023-04-27 01:56:22,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/994be0785ebb492a95508800e1e6850f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/994be0785ebb492a95508800e1e6850f 2023-04-27 01:56:22,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/994be0785ebb492a95508800e1e6850f, entries=150, sequenceid=157, filesize=11.8 K 2023-04-27 01:56:22,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/bc777593ad314f8885e16e16b0686b08 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bc777593ad314f8885e16e16b0686b08 2023-04-27 01:56:22,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bc777593ad314f8885e16e16b0686b08, entries=150, sequenceid=157, filesize=11.8 K 2023-04-27 01:56:22,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ec4ba6d393d7d8e622c952e9d8a3e627 in 468ms, sequenceid=157, compaction requested=true 2023-04-27 01:56:22,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:22,302 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:22,302 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:22,303 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59902 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:22,303 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143186 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:22,303 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/A is initiating minor compaction (all files) 2023-04-27 01:56:22,303 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/A in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,303 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c3707fe8ddd543be942337739b1c7ed5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/de021e00e84c4301a10cf5a53ab74c14, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/48762217f78b4ee2930397a107f09e47, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/08f78cf8d1da4653beebdc287a91efa3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7469efaa34464e168babea6a03f5e1d5] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=139.8 K 2023-04-27 01:56:22,303 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/B is initiating minor compaction (all files) 2023-04-27 01:56:22,304 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/B in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,304 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/94cf8c6aa72949f5a1ee4ac98a4bcf7d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3dcbf3f4cea24145a45360dcd0c271b3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/4fb1e27e6e7e46a791cc7a41a2783012, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9b80c85f2e134048af6c5698ab716e87, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/994be0785ebb492a95508800e1e6850f] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=58.5 K 2023-04-27 01:56:22,304 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c3707fe8ddd543be942337739b1c7ed5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682560578315 2023-04-27 01:56:22,304 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 94cf8c6aa72949f5a1ee4ac98a4bcf7d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682560578315 2023-04-27 01:56:22,304 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting de021e00e84c4301a10cf5a53ab74c14, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682560578329 2023-04-27 01:56:22,304 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3dcbf3f4cea24145a45360dcd0c271b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682560578329 2023-04-27 01:56:22,304 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 48762217f78b4ee2930397a107f09e47, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1682560578948 2023-04-27 01:56:22,305 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 4fb1e27e6e7e46a791cc7a41a2783012, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1682560578948 2023-04-27 01:56:22,305 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 08f78cf8d1da4653beebdc287a91efa3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682560579585 2023-04-27 01:56:22,305 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9b80c85f2e134048af6c5698ab716e87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682560579585 2023-04-27 01:56:22,305 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 7469efaa34464e168babea6a03f5e1d5, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682560580715 2023-04-27 01:56:22,305 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 994be0785ebb492a95508800e1e6850f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682560580716 2023-04-27 01:56:22,313 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#B#compaction#528 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:22,322 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#A#compaction#529 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:22,327 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/197957d464394a9280f3a6e640874590 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/197957d464394a9280f3a6e640874590 2023-04-27 01:56:22,333 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/B of ec4ba6d393d7d8e622c952e9d8a3e627 into 197957d464394a9280f3a6e640874590(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:22,333 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:22,333 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/B, priority=11, startTime=1682560582302; duration=0sec 2023-04-27 01:56:22,333 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:22,333 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:22,335 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59902 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:22,335 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/C is initiating minor compaction (all files) 2023-04-27 01:56:22,335 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/C in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,335 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/56ac11ea5da24837a23ba2b1125b0825, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1f863289a6bb4f45ab3ce62df57709b5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/ef21d999337641e9be103e80ef625072, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3de56c46de744e1d99bf601d3230e050, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bc777593ad314f8885e16e16b0686b08] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=58.5 K 2023-04-27 01:56:22,335 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 56ac11ea5da24837a23ba2b1125b0825, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682560578315 2023-04-27 01:56:22,335 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1f863289a6bb4f45ab3ce62df57709b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682560578329 2023-04-27 01:56:22,336 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ef21d999337641e9be103e80ef625072, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1682560578948 2023-04-27 01:56:22,336 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3de56c46de744e1d99bf601d3230e050, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682560579585 2023-04-27 01:56:22,336 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting bc777593ad314f8885e16e16b0686b08, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682560580716 2023-04-27 01:56:22,343 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#C#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:22,389 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:22,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:22,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:22,393 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:22,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:22,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:22,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,395 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:22,395 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,396 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,396 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:22,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:22,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:22,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:22,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:22,400 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:22,400 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:22,400 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:56:22,400 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:22,400 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:22,401 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:22,401 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:22,401 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:22,401 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:22,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:22,458 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:22,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560642472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682560582468, max timestamp=1682560582475Num uniques 859; 2023-04-27 01:56:22,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682560582468, max timestamp=1682560582475Num uniques 859; 2023-04-27 01:56:22,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 01:56:22,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682560582468, max timestamp=1682560582475Num uniques 859; 2023-04-27 01:56:22,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 01:56:22,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 01:56:22,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560642489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560642513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560642513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560642513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560642578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560642614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560642618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560642618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560642618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,759 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/6362e932a3264fb8b27c7e11bdd3a5a7 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6362e932a3264fb8b27c7e11bdd3a5a7 2023-04-27 01:56:22,763 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/C of ec4ba6d393d7d8e622c952e9d8a3e627 into 6362e932a3264fb8b27c7e11bdd3a5a7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:22,763 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:22,763 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/C, priority=11, startTime=1682560582302; duration=0sec 2023-04-27 01:56:22,763 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:22,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560642781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,813 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e10cddcdfeed479fba9346bf38fda714 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e10cddcdfeed479fba9346bf38fda714 2023-04-27 01:56:22,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560642818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560642821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560642821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:22,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:22,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560642822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560643085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560643127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560643127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560643127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560643128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,142 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/7660e69c52194df7885a0bfbafb82081 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7660e69c52194df7885a0bfbafb82081 2023-04-27 01:56:23,147 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/A of ec4ba6d393d7d8e622c952e9d8a3e627 into 7660e69c52194df7885a0bfbafb82081(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:23,147 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:23,147 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/A, priority=11, startTime=1682560582302; duration=0sec 2023-04-27 01:56:23,147 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:23,219 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/811818fbf9074aed8439491fdf6bcca8 2023-04-27 01:56:23,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560643591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560643630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560643632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560643632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:23,635 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/843da33b29654b998fb39c4cff712c25 2023-04-27 01:56:23,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:23,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560643635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,045 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/cd9d305721aa472aa21441a68c4f900d 2023-04-27 01:56:24,049 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/811818fbf9074aed8439491fdf6bcca8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/811818fbf9074aed8439491fdf6bcca8 2023-04-27 01:56:24,053 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/811818fbf9074aed8439491fdf6bcca8, entries=150, sequenceid=169, filesize=25.3 K 2023-04-27 01:56:24,054 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/843da33b29654b998fb39c4cff712c25 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/843da33b29654b998fb39c4cff712c25 2023-04-27 01:56:24,057 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/843da33b29654b998fb39c4cff712c25, entries=150, sequenceid=169, filesize=11.8 K 2023-04-27 01:56:24,058 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/cd9d305721aa472aa21441a68c4f900d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/cd9d305721aa472aa21441a68c4f900d 2023-04-27 01:56:24,061 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/cd9d305721aa472aa21441a68c4f900d, entries=150, sequenceid=169, filesize=11.8 K 2023-04-27 01:56:24,062 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for ec4ba6d393d7d8e622c952e9d8a3e627 in 1662ms, sequenceid=169, compaction requested=false 2023-04-27 01:56:24,062 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:24,062 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:24,072 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,072 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:24,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43cb0b9d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:24,074 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:24,074 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:24,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:24,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:24,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,077 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:24,078 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:24,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,078 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:24,080 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:24,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:24,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:24,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:24,080 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:24,082 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,082 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,082 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,082 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,086 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,086 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:24,086 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:24,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:24,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:24,086 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:24,086 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:24,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:24,186 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:24,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:24,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:24,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:24,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:24,190 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:24,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:24,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:24,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:24,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:24,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:24,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:24,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:24,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:24,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:24,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:24,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:24,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:24,194 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,194 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:24,194 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-27 01:56:24,195 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:24,195 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:24,195 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:24,195 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:24,195 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:24,195 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 01:56:24,217 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304278403a3eb203a4e6b9839b12341dbf25e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278403a3eb203a4e6b9839b12341dbf25e 2023-04-27 01:56:24,224 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/d39e58ba597b4c3ba4b44953e97c9af0 2023-04-27 01:56:24,234 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/9bf8a1bd81044bd095570c13b2325462 2023-04-27 01:56:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:24,599 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:24,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:24,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43496 deadline: 1682560644619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560644636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560644638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560644642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,647 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/85909ff8aaf3424299fea1aa46ffc5e5 2023-04-27 01:56:24,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:24,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560644647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,651 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/d39e58ba597b4c3ba4b44953e97c9af0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d39e58ba597b4c3ba4b44953e97c9af0 2023-04-27 01:56:24,655 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d39e58ba597b4c3ba4b44953e97c9af0, entries=150, sequenceid=206, filesize=25.3 K 2023-04-27 01:56:24,655 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/9bf8a1bd81044bd095570c13b2325462 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9bf8a1bd81044bd095570c13b2325462 2023-04-27 01:56:24,659 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9bf8a1bd81044bd095570c13b2325462, entries=150, sequenceid=206, filesize=11.8 K 2023-04-27 01:56:24,659 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/85909ff8aaf3424299fea1aa46ffc5e5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/85909ff8aaf3424299fea1aa46ffc5e5 2023-04-27 01:56:24,662 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/85909ff8aaf3424299fea1aa46ffc5e5, entries=150, sequenceid=206, filesize=11.8 K 2023-04-27 01:56:24,663 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for ec4ba6d393d7d8e622c952e9d8a3e627 in 469ms, sequenceid=206, compaction requested=true 2023-04-27 01:56:24,663 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:24,663 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:24,668 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:24,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34da818b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:24,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:24,670 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,671 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:24,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:24,671 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:24,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,671 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:24,672 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,673 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:24,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:24,675 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:24,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:24,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:24,675 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:24,675 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:24,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,680 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,684 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:24,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:24,684 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:24,685 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:24,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:24,688 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:24,688 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:24,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:24,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 01:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:24,740 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042744ffea9f396c4dcf90a0182c367bb3da to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042744ffea9f396c4dcf90a0182c367bb3da 2023-04-27 01:56:24,745 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=217, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/595303c23a80426b89dfb16cc3a4d827 2023-04-27 01:56:24,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/7cc6d465fced49e8aff4a655c25ea5cb 2023-04-27 01:56:24,764 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3cabdad74a9740159260149f9a3d77bc 2023-04-27 01:56:24,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/595303c23a80426b89dfb16cc3a4d827 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/595303c23a80426b89dfb16cc3a4d827 2023-04-27 01:56:24,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/595303c23a80426b89dfb16cc3a4d827, entries=150, sequenceid=217, filesize=25.3 K 2023-04-27 01:56:24,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/7cc6d465fced49e8aff4a655c25ea5cb as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/7cc6d465fced49e8aff4a655c25ea5cb 2023-04-27 01:56:24,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/7cc6d465fced49e8aff4a655c25ea5cb, entries=100, sequenceid=217, filesize=9.4 K 2023-04-27 01:56:24,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3cabdad74a9740159260149f9a3d77bc as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3cabdad74a9740159260149f9a3d77bc 2023-04-27 01:56:24,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3cabdad74a9740159260149f9a3d77bc, entries=100, sequenceid=217, filesize=9.4 K 2023-04-27 01:56:24,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=93.93 KB/96180 for ec4ba6d393d7d8e622c952e9d8a3e627 in 52ms, sequenceid=217, compaction requested=true 2023-04-27 01:56:24,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:24,779 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:24,779 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:24,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:24,780 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:24,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:24,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:56:24,781 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103922 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:24,781 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:24,781 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/A is initiating minor compaction (all files) 2023-04-27 01:56:24,781 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:24,781 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/A in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,781 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:24,781 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7660e69c52194df7885a0bfbafb82081, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/811818fbf9074aed8439491fdf6bcca8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d39e58ba597b4c3ba4b44953e97c9af0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/595303c23a80426b89dfb16cc3a4d827] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=101.5 K 2023-04-27 01:56:24,781 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:24,781 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46092 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:24,781 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/B is initiating minor compaction (all files) 2023-04-27 01:56:24,781 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/B in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,781 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 7660e69c52194df7885a0bfbafb82081, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682560580716 2023-04-27 01:56:24,781 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:24,781 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/197957d464394a9280f3a6e640874590, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/843da33b29654b998fb39c4cff712c25, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9bf8a1bd81044bd095570c13b2325462, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/7cc6d465fced49e8aff4a655c25ea5cb] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=45.0 K 2023-04-27 01:56:24,781 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:24,782 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 811818fbf9074aed8439491fdf6bcca8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682560581841 2023-04-27 01:56:24,782 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 197957d464394a9280f3a6e640874590, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682560580716 2023-04-27 01:56:24,783 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 843da33b29654b998fb39c4cff712c25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682560581841 2023-04-27 01:56:24,783 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting d39e58ba597b4c3ba4b44953e97c9af0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682560582479 2023-04-27 01:56:24,783 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9bf8a1bd81044bd095570c13b2325462, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682560582479 2023-04-27 01:56:24,783 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 595303c23a80426b89dfb16cc3a4d827, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682560584613 2023-04-27 01:56:24,783 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 7cc6d465fced49e8aff4a655c25ea5cb, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682560584613 2023-04-27 01:56:24,788 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:24,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:24,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:24,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:24,792 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:24,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:24,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:24,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:24,792 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,793 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,794 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:24,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:24,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,794 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:24,801 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#B#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:24,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:24,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:24,812 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,812 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:24,812 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:24,812 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:24,817 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#A#compaction#541 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:24,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@927face[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:24,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:24,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:24,822 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,822 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:24,822 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427724bea67ec484b38af6234c41a6c3c9b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427724bea67ec484b38af6234c41a6c3c9b 2023-04-27 01:56:24,823 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:24,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560644821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:24,826 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:24,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:24,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:24,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:24,826 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:24,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,830 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,832 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,833 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:24,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:24,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,833 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:24,833 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:24,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,833 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,833 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/3388cfaea35743dd86e1d20718a9f9b2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3388cfaea35743dd86e1d20718a9f9b2 2023-04-27 01:56:24,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,835 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=235, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/00cc855220c24fba9529b80e2ce44d92 2023-04-27 01:56:24,839 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/B of ec4ba6d393d7d8e622c952e9d8a3e627 into 3388cfaea35743dd86e1d20718a9f9b2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:24,839 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:24,839 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/B, priority=12, startTime=1682560584780; duration=0sec 2023-04-27 01:56:24,839 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:24,839 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:24,840 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46092 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:24,840 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/C is initiating minor compaction (all files) 2023-04-27 01:56:24,841 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/C in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,841 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6362e932a3264fb8b27c7e11bdd3a5a7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/cd9d305721aa472aa21441a68c4f900d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/85909ff8aaf3424299fea1aa46ffc5e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3cabdad74a9740159260149f9a3d77bc] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=45.0 K 2023-04-27 01:56:24,841 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6362e932a3264fb8b27c7e11bdd3a5a7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682560580716 2023-04-27 01:56:24,841 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting cd9d305721aa472aa21441a68c4f900d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682560581841 2023-04-27 01:56:24,842 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 85909ff8aaf3424299fea1aa46ffc5e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682560582479 2023-04-27 01:56:24,843 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3cabdad74a9740159260149f9a3d77bc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682560584613 2023-04-27 01:56:24,857 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#C#compaction#544 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:24,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:24,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560644927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,933 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:24,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:24,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:24,936 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:24,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:24,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:24,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:24,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,937 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:24,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:24,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:24,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:24,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:24,941 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,941 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:24,941 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:24,941 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:24,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c77fe6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:24,945 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:24,945 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,946 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:24,948 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:24,951 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:24,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:24,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:24,951 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:24,951 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:24,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,954 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:24,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:24,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:24,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:24,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:24,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:24,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:24,958 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:24,959 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:24,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:24,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:24,959 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:24,959 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:24,959 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:24,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:25,059 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:25,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:25,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:25,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:25,063 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:25,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:25,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:25,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:25,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,063 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,064 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:25,064 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,065 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:25,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:25,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:25,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:25,066 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:25,066 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:25,066 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:25,066 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:25,068 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:25,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:25,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4b1b80[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:25,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:25,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,075 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:25,075 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,075 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:25,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:25,076 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:25,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:25,078 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:25,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:25,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:25,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:25,079 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:25,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:25,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,080 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:25,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:25,081 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,082 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:25,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,085 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:25,085 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:25,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:25,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:25,086 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:25,086 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:25,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:25,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:25,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560645130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,186 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:25,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:25,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:25,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:25,190 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:25,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:25,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:25,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:25,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,190 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,191 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:25,191 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:25,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:25,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:25,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:25,193 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:25,194 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:25,194 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:25,194 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:25,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bfd07bb[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:25,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:25,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:25,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:25,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:25,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:25,201 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:25,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:25,203 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:25,203 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:25,203 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:25,203 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:25,203 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:25,205 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,205 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,205 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,205 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:25,207 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:25,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:25,208 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:25,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:25,209 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:25,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:25,209 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:25,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:25,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:25,209 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,209 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:25,209 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:25,209 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:25,209 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,257 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/93b9fcd31ecb4920aad1033d7ccf0a3a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/93b9fcd31ecb4920aad1033d7ccf0a3a 2023-04-27 01:56:25,261 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/A of ec4ba6d393d7d8e622c952e9d8a3e627 into 93b9fcd31ecb4920aad1033d7ccf0a3a(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:25,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/877c33927f4148cb8d94b2ea63d1daee 2023-04-27 01:56:25,261 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:25,262 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/A, priority=12, startTime=1682560584779; duration=0sec 2023-04-27 01:56:25,262 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:25,270 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/946d3d6abd074c0c87d16e7dfc971530 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/946d3d6abd074c0c87d16e7dfc971530 2023-04-27 01:56:25,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/0de5382f689d4babafc99483a2636360 2023-04-27 01:56:25,276 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/C of ec4ba6d393d7d8e622c952e9d8a3e627 into 946d3d6abd074c0c87d16e7dfc971530(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:25,276 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:25,276 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/C, priority=12, startTime=1682560584780; duration=0sec 2023-04-27 01:56:25,276 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:25,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/00cc855220c24fba9529b80e2ce44d92 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/00cc855220c24fba9529b80e2ce44d92 2023-04-27 01:56:25,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/00cc855220c24fba9529b80e2ce44d92, entries=200, sequenceid=235, filesize=32.1 K 2023-04-27 01:56:25,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/877c33927f4148cb8d94b2ea63d1daee as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/877c33927f4148cb8d94b2ea63d1daee 2023-04-27 01:56:25,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/877c33927f4148cb8d94b2ea63d1daee, entries=150, sequenceid=235, filesize=11.8 K 2023-04-27 01:56:25,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/0de5382f689d4babafc99483a2636360 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0de5382f689d4babafc99483a2636360 2023-04-27 01:56:25,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0de5382f689d4babafc99483a2636360, entries=150, sequenceid=235, filesize=11.8 K 2023-04-27 01:56:25,290 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ec4ba6d393d7d8e622c952e9d8a3e627 in 510ms, sequenceid=235, compaction requested=false 2023-04-27 01:56:25,290 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:25,309 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:25,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:25,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:25,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:25,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:25,313 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:25,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:25,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:25,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:25,313 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,314 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,315 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:25,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,315 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:25,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:25,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:25,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:25,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:25,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:25,316 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:25,317 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:25,317 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:25,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:25,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:25,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:25,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560645477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:25,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560645581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:25,741 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427bb9d1c0040ee4136ba2e6efa5cb119fb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bb9d1c0040ee4136ba2e6efa5cb119fb 2023-04-27 01:56:25,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560645783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560646087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,146 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=252, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/36a93504731145a7953a328636071ed2 2023-04-27 01:56:26,156 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/d75c97259cc64e018f704214056c75b8 2023-04-27 01:56:26,167 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/c4b4d72f61a34b909248317b781a4d51 2023-04-27 01:56:26,171 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/36a93504731145a7953a328636071ed2 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/36a93504731145a7953a328636071ed2 2023-04-27 01:56:26,175 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/36a93504731145a7953a328636071ed2, entries=150, sequenceid=252, filesize=25.3 K 2023-04-27 01:56:26,176 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/d75c97259cc64e018f704214056c75b8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/d75c97259cc64e018f704214056c75b8 2023-04-27 01:56:26,179 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/d75c97259cc64e018f704214056c75b8, entries=150, sequenceid=252, filesize=11.8 K 2023-04-27 01:56:26,180 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/c4b4d72f61a34b909248317b781a4d51 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/c4b4d72f61a34b909248317b781a4d51 2023-04-27 01:56:26,183 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/c4b4d72f61a34b909248317b781a4d51, entries=150, sequenceid=252, filesize=11.8 K 2023-04-27 01:56:26,184 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ec4ba6d393d7d8e622c952e9d8a3e627 in 867ms, sequenceid=252, compaction requested=true 2023-04-27 01:56:26,184 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:26,184 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:26,187 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3a6380[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:26,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:26,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,190 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:26,192 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:26,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:26,193 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:26,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:26,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:26,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:26,193 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:26,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:26,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,195 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:26,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:26,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:26,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:26,202 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:26,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:26,202 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:26,203 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:26,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:26,203 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:26,303 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:26,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:26,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:26,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:26,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:26,306 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:26,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:26,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:26,306 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:26,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,307 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:26,308 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:26,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:26,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,308 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:26,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:26,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:26,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:26,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:26,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:26,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:26,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:26,311 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:56:26,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:26,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:26,311 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:26,312 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:26,312 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:26,312 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:26,334 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427d54906720a6a4ba985d1a1fa96e3bbca to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d54906720a6a4ba985d1a1fa96e3bbca 2023-04-27 01:56:26,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:26,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:26,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560646623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560646643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,649 DEBUG [Thread-3734] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:26,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560646651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,654 DEBUG [Thread-3732] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:26,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560646655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,658 DEBUG [Thread-3726] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:26,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560646662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,665 DEBUG [Thread-3728] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:26,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560646727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:26,739 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/4ef60a7db79546e7b0301c65a42e0d90 2023-04-27 01:56:26,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:26,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560646931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,149 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/86b7ad9c6a2744ada258b34ceddebfd6 2023-04-27 01:56:27,158 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3e48432f10934a9b9e2c98b4abdefb39 2023-04-27 01:56:27,162 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/4ef60a7db79546e7b0301c65a42e0d90 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4ef60a7db79546e7b0301c65a42e0d90 2023-04-27 01:56:27,165 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4ef60a7db79546e7b0301c65a42e0d90, entries=150, sequenceid=270, filesize=25.4 K 2023-04-27 01:56:27,165 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/86b7ad9c6a2744ada258b34ceddebfd6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/86b7ad9c6a2744ada258b34ceddebfd6 2023-04-27 01:56:27,169 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/86b7ad9c6a2744ada258b34ceddebfd6, entries=150, sequenceid=270, filesize=11.9 K 2023-04-27 01:56:27,169 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3e48432f10934a9b9e2c98b4abdefb39 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3e48432f10934a9b9e2c98b4abdefb39 2023-04-27 01:56:27,172 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3e48432f10934a9b9e2c98b4abdefb39, entries=150, sequenceid=270, filesize=11.9 K 2023-04-27 01:56:27,173 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ec4ba6d393d7d8e622c952e9d8a3e627 in 862ms, sequenceid=270, compaction requested=true 2023-04-27 01:56:27,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:27,173 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:27,174 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76b90d8e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:27,176 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:27,177 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,178 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:27,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:27,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,179 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:27,179 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,181 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:27,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:27,189 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:27,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:27,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:27,189 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:27,189 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:27,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,192 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,193 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,193 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,194 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:27,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:27,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,199 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:27,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:27,198 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,199 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:27,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:27,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 01:56:27,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:27,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:27,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:27,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:27,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:27,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:27,251 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042750e13cc9255e4a98a40030ae5addf33c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042750e13cc9255e4a98a40030ae5addf33c 2023-04-27 01:56:27,255 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/cdb7b428b39e4da4a16b4135ae847ded 2023-04-27 01:56:27,299 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:27,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:27,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:27,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:27,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:27,302 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:27,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:27,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:27,302 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,303 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,304 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,304 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:27,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:27,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:27,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:27,305 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:27,308 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560647307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:27,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ed8fc2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:27,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:27,310 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,311 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:27,311 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,312 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:27,313 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:27,316 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:27,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:27,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:27,316 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:27,316 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,318 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,318 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,320 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:27,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:27,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:27,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,323 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,323 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,323 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:27,324 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560647411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,424 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:27,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:27,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:27,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:27,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:27,430 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:27,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:27,430 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:27,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,431 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,432 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,432 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:27,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:27,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:27,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:27,433 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:27,434 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,434 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:27,434 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:27,434 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:27,434 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:27,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,436 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,436 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:27,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263e861b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:27,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:27,438 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,438 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,439 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:27,441 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:27,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,441 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:27,444 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:27,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:27,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:27,444 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:27,444 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:27,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,446 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,447 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,447 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,449 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,451 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,452 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,452 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,452 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,451 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:27,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,452 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:27,452 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:27,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,552 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:27,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:27,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:27,556 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:27,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:27,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:27,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,556 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,557 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,557 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,558 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:27,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:27,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:27,559 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,559 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:27,559 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:27,560 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:27,560 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:27,562 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:27,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3806d73f[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:27,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:27,564 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,565 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:27,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:27,565 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:27,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,565 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:27,566 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:27,569 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:27,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:27,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:27,569 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:27,569 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,570 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,571 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,572 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:27,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:27,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:27,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,578 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:27,578 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:27,578 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,578 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,579 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:27,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560647614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/b18c3246154343059c353e3412ade4da 2023-04-27 01:56:27,678 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:27,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:27,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:27,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:27,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:27,681 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:27,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:27,681 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:27,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,682 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,683 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,683 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:27,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:27,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:27,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:27,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:27,685 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:27,686 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,686 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,687 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:27,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d68cc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:27,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:27,688 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,690 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,691 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:27,691 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:27,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,691 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:27,691 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,692 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:27,695 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:27,695 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:27,695 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:27,695 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:27,695 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:27,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,696 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,697 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,698 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:27,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:27,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:27,700 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:27,701 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:27,700 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,700 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,701 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,701 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,801 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:27,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:27,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:27,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:27,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:27,804 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:27,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:27,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:27,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,811 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,812 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,813 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:27,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:27,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:27,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:27,814 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,814 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:27,814 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:27,814 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:27,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,816 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f27645c[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:27,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:27,817 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,818 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:27,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,819 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:27,820 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:27,823 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:27,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:27,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:27,823 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:27,823 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:27,825 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,825 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,825 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,826 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,830 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:27,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:27,830 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:27,830 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,830 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:27,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560647917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,930 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:27,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:27,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:27,934 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:27,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:27,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:27,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,934 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,936 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,936 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:27,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:27,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:27,938 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,938 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:27,938 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:27,938 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:27,939 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,939 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:27,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d7b2ffd[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:27,941 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:27,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,943 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:27,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:27,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:27,944 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,944 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:27,945 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:27,945 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,945 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,945 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:27,947 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:27,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:27,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:27,947 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:27,948 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:27,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,950 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:27,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,956 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,957 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:27,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:27,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:27,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:27,964 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:27,964 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:27,964 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:27,964 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:27,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:27,965 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,064 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:28,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:28,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:28,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:28,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:28,068 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:28,068 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,069 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,070 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:28,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:28,073 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,073 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:28,073 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:28,073 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,075 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/321f760edf0349f6b13c65da75c31e59 2023-04-27 01:56:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34aae42[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:28,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:28,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/cdb7b428b39e4da4a16b4135ae847ded as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/cdb7b428b39e4da4a16b4135ae847ded 2023-04-27 01:56:28,081 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:28,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:28,081 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:28,081 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,081 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:28,083 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/cdb7b428b39e4da4a16b4135ae847ded, entries=200, sequenceid=285, filesize=32.3 K 2023-04-27 01:56:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/b18c3246154343059c353e3412ade4da as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b18c3246154343059c353e3412ade4da 2023-04-27 01:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:28,086 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:28,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:28,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:28,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:28,086 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:28,088 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,088 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,088 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,088 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:28,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b18c3246154343059c353e3412ade4da, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 01:56:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/321f760edf0349f6b13c65da75c31e59 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/321f760edf0349f6b13c65da75c31e59 2023-04-27 01:56:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,090 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,092 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,092 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,092 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,092 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:28,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:28,092 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,093 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:28,093 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:28,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:28,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,093 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:28,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/321f760edf0349f6b13c65da75c31e59, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 01:56:28,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ec4ba6d393d7d8e622c952e9d8a3e627 in 857ms, sequenceid=285, compaction requested=true 2023-04-27 01:56:28,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:28,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:28,095 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:28,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:28,095 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:28,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:28,096 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144238 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:28,097 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/A is initiating minor compaction (all files) 2023-04-27 01:56:28,097 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/A in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,097 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60908 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:28,097 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/93b9fcd31ecb4920aad1033d7ccf0a3a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/00cc855220c24fba9529b80e2ce44d92, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/36a93504731145a7953a328636071ed2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4ef60a7db79546e7b0301c65a42e0d90, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/cdb7b428b39e4da4a16b4135ae847ded] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=140.9 K 2023-04-27 01:56:28,097 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/B is initiating minor compaction (all files) 2023-04-27 01:56:28,097 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/B in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,097 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3388cfaea35743dd86e1d20718a9f9b2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/877c33927f4148cb8d94b2ea63d1daee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/d75c97259cc64e018f704214056c75b8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/86b7ad9c6a2744ada258b34ceddebfd6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b18c3246154343059c353e3412ade4da] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=59.5 K 2023-04-27 01:56:28,097 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 93b9fcd31ecb4920aad1033d7ccf0a3a, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682560582489 2023-04-27 01:56:28,098 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 00cc855220c24fba9529b80e2ce44d92, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682560584768 2023-04-27 01:56:28,098 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3388cfaea35743dd86e1d20718a9f9b2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682560582489 2023-04-27 01:56:28,098 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 36a93504731145a7953a328636071ed2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682560584812 2023-04-27 01:56:28,098 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 877c33927f4148cb8d94b2ea63d1daee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682560584772 2023-04-27 01:56:28,098 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 4ef60a7db79546e7b0301c65a42e0d90, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682560585437 2023-04-27 01:56:28,098 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting d75c97259cc64e018f704214056c75b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682560584812 2023-04-27 01:56:28,099 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 86b7ad9c6a2744ada258b34ceddebfd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682560585437 2023-04-27 01:56:28,099 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting cdb7b428b39e4da4a16b4135ae847ded, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682560586609 2023-04-27 01:56:28,099 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b18c3246154343059c353e3412ade4da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682560586619 2023-04-27 01:56:28,107 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#B#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:28,118 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#A#compaction#556 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:28,125 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/9f32e4e2bc9f49f0a6e484677c724ae4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9f32e4e2bc9f49f0a6e484677c724ae4 2023-04-27 01:56:28,129 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/B of ec4ba6d393d7d8e622c952e9d8a3e627 into 9f32e4e2bc9f49f0a6e484677c724ae4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:28,129 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:28,129 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/B, priority=11, startTime=1682560588095; duration=0sec 2023-04-27 01:56:28,129 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:28,129 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:28,131 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60908 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:28,131 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/C is initiating minor compaction (all files) 2023-04-27 01:56:28,131 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/C in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,131 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/946d3d6abd074c0c87d16e7dfc971530, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0de5382f689d4babafc99483a2636360, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/c4b4d72f61a34b909248317b781a4d51, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3e48432f10934a9b9e2c98b4abdefb39, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/321f760edf0349f6b13c65da75c31e59] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=59.5 K 2023-04-27 01:56:28,131 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 946d3d6abd074c0c87d16e7dfc971530, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682560582489 2023-04-27 01:56:28,132 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 0de5382f689d4babafc99483a2636360, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682560584772 2023-04-27 01:56:28,132 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c4b4d72f61a34b909248317b781a4d51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682560584812 2023-04-27 01:56:28,132 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3e48432f10934a9b9e2c98b4abdefb39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1682560585437 2023-04-27 01:56:28,133 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 321f760edf0349f6b13c65da75c31e59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682560586619 2023-04-27 01:56:28,146 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#C#compaction#557 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:28,162 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/be01351e978145a18c38c846abcb9d58 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/be01351e978145a18c38c846abcb9d58 2023-04-27 01:56:28,162 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/e683e5be33784b28b00cee320114921e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e683e5be33784b28b00cee320114921e 2023-04-27 01:56:28,166 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/A of ec4ba6d393d7d8e622c952e9d8a3e627 into e683e5be33784b28b00cee320114921e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:28,166 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:28,166 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/A, priority=11, startTime=1682560588095; duration=0sec 2023-04-27 01:56:28,166 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:28,166 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/C of ec4ba6d393d7d8e622c952e9d8a3e627 into be01351e978145a18c38c846abcb9d58(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:28,166 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:28,166 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/C, priority=11, startTime=1682560588095; duration=0sec 2023-04-27 01:56:28,166 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:28,193 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:28,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:28,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:28,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:28,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:28,197 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:28,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:28,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:28,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,199 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,199 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:28,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:28,201 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 01:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:28,201 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:28,202 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:28,217 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427b7b4524fa37f4692b3bef2ed33a59a2e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b7b4524fa37f4692b3bef2ed33a59a2e 2023-04-27 01:56:28,222 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/27313dc531c24da29357fcd8c1e1f231 2023-04-27 01:56:28,240 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/c8ea7b13dc8241229374968a3e6bd5f4 2023-04-27 01:56:28,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:28,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:28,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:28,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560648451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:28,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560648554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,666 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/fea8c39711fd4e44b6b8d77e27635bc1 2023-04-27 01:56:28,672 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/27313dc531c24da29357fcd8c1e1f231 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/27313dc531c24da29357fcd8c1e1f231 2023-04-27 01:56:28,676 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/27313dc531c24da29357fcd8c1e1f231, entries=150, sequenceid=305, filesize=25.4 K 2023-04-27 01:56:28,677 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/c8ea7b13dc8241229374968a3e6bd5f4 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/c8ea7b13dc8241229374968a3e6bd5f4 2023-04-27 01:56:28,681 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/c8ea7b13dc8241229374968a3e6bd5f4, entries=150, sequenceid=305, filesize=11.9 K 2023-04-27 01:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/fea8c39711fd4e44b6b8d77e27635bc1 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/fea8c39711fd4e44b6b8d77e27635bc1 2023-04-27 01:56:28,687 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/fea8c39711fd4e44b6b8d77e27635bc1, entries=150, sequenceid=305, filesize=11.9 K 2023-04-27 01:56:28,688 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ec4ba6d393d7d8e622c952e9d8a3e627 in 487ms, sequenceid=305, compaction requested=false 2023-04-27 01:56:28,688 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:28,688 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:28,689 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,689 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e0c8def[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:28,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:28,692 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,693 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:28,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,694 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:28,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:28,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,695 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:28,695 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:28,699 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:28,699 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:28,699 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:28,699 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:28,699 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:28,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,705 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,706 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,707 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,710 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:28,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,710 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:28,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,710 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:28,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:28,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 01:56:28,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:28,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:28,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:28,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:28,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:28,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:28,770 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427b1443f87123d463296578a69c8fe589f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b1443f87123d463296578a69c8fe589f 2023-04-27 01:56:28,775 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/b830f719a30f4a8b9572a8fc90c81b3c 2023-04-27 01:56:28,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:28,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560648795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,811 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:28,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:28,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:28,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:28,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:28,814 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:28,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:28,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:28,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,814 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,815 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,815 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:28,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:28,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:28,817 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,817 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:28,817 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:28,817 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:28,819 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aed01f6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:28,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:28,820 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,822 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:28,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,824 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:28,824 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:28,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,824 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:28,826 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:28,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:28,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:28,827 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:28,827 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:28,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,828 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,829 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,832 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,835 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:28,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,835 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:28,835 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:28,835 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:28,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:28,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:28,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560648898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,935 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:28,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:28,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:28,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:28,939 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:28,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:28,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:28,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,939 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,942 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,942 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:28,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:28,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:28,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:28,944 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:28,948 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,949 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebdd391[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:28,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:28,950 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,951 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:28,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:28,951 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:28,952 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,952 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:28,953 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:28,955 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:28,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:28,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:28,955 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:28,955 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:28,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,960 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:28,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:28,962 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,963 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:28,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:28,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:28,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:28,966 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:28,966 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:28,966 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:28,966 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:28,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:29,066 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:29,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:29,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:29,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:29,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:29,070 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:29,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:29,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:29,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,070 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,072 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,072 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,072 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:29,072 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:29,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:29,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:29,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:29,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:29,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,074 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:29,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:29,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:29,075 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:29,076 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,077 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d34eba[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:29,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:29,078 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,079 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,079 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:29,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,080 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:29,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:29,080 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:29,082 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:29,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:29,082 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:29,084 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:29,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:29,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:29,084 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:29,084 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:29,086 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,086 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,086 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,086 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,087 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,089 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:29,090 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:29,089 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:29,090 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:29,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:29,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:29,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:29,090 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,090 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:29,090 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:29,090 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:29,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560649102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/e11f4003e42a4458a4948c9c63489e9d 2023-04-27 01:56:29,190 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:29,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:29,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/606b2c3ce38c4508824a43237c82d588 2023-04-27 01:56:29,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:29,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:29,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:29,195 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:29,195 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,195 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:29,195 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:29,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,196 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,197 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,197 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:29,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:29,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,198 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:29,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/b830f719a30f4a8b9572a8fc90c81b3c as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b830f719a30f4a8b9572a8fc90c81b3c 2023-04-27 01:56:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:29,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:29,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:29,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,199 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:29,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:29,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:29,200 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:29,200 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:29,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b830f719a30f4a8b9572a8fc90c81b3c, entries=200, sequenceid=321, filesize=32.3 K 2023-04-27 01:56:29,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/e11f4003e42a4458a4948c9c63489e9d as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11f4003e42a4458a4948c9c63489e9d 2023-04-27 01:56:29,203 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bac9ba[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:29,205 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:29,206 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,207 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11f4003e42a4458a4948c9c63489e9d, entries=150, sequenceid=321, filesize=11.9 K 2023-04-27 01:56:29,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:29,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:29,208 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:29,208 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,208 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:29,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/606b2c3ce38c4508824a43237c82d588 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/606b2c3ce38c4508824a43237c82d588 2023-04-27 01:56:29,209 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:29,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,209 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/606b2c3ce38c4508824a43237c82d588, entries=150, sequenceid=321, filesize=11.9 K 2023-04-27 01:56:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:29,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:29,212 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:29,212 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:29,212 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:29,212 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:29,212 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:29,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ec4ba6d393d7d8e622c952e9d8a3e627 in 455ms, sequenceid=321, compaction requested=true 2023-04-27 01:56:29,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:29,213 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:29,213 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:29,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:29,214 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,214 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,214 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,214 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:29,214 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:29,215 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/A is initiating minor compaction (all files) 2023-04-27 01:56:29,215 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/B is initiating minor compaction (all files) 2023-04-27 01:56:29,215 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/A in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,215 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/B in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,215 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9f32e4e2bc9f49f0a6e484677c724ae4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/c8ea7b13dc8241229374968a3e6bd5f4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11f4003e42a4458a4948c9c63489e9d] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=36.3 K 2023-04-27 01:56:29,215 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e683e5be33784b28b00cee320114921e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/27313dc531c24da29357fcd8c1e1f231, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b830f719a30f4a8b9572a8fc90c81b3c] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=83.8 K 2023-04-27 01:56:29,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:29,215 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,215 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 9f32e4e2bc9f49f0a6e484677c724ae4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682560586619 2023-04-27 01:56:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,216 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e683e5be33784b28b00cee320114921e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682560586619 2023-04-27 01:56:29,216 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting c8ea7b13dc8241229374968a3e6bd5f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1682560587295 2023-04-27 01:56:29,216 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 27313dc531c24da29357fcd8c1e1f231, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1682560587295 2023-04-27 01:56:29,217 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting e11f4003e42a4458a4948c9c63489e9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682560588447 2023-04-27 01:56:29,217 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting b830f719a30f4a8b9572a8fc90c81b3c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682560588442 2023-04-27 01:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,218 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:29,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,222 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:29,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:29,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:29,222 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:29,223 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:29,223 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:29,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:29,225 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#B#compaction#564 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:29,237 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#A#compaction#565 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:29,243 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/78a5b88772d34a5ebecf6c7d88410964 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/78a5b88772d34a5ebecf6c7d88410964 2023-04-27 01:56:29,248 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/B of ec4ba6d393d7d8e622c952e9d8a3e627 into 78a5b88772d34a5ebecf6c7d88410964(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:29,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:29,248 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/B, priority=13, startTime=1682560589213; duration=0sec 2023-04-27 01:56:29,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:29,248 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 01:56:29,249 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 01:56:29,249 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/C is initiating minor compaction (all files) 2023-04-27 01:56:29,249 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/C in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,249 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/be01351e978145a18c38c846abcb9d58, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/fea8c39711fd4e44b6b8d77e27635bc1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/606b2c3ce38c4508824a43237c82d588] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=36.3 K 2023-04-27 01:56:29,250 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting be01351e978145a18c38c846abcb9d58, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682560586619 2023-04-27 01:56:29,250 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting fea8c39711fd4e44b6b8d77e27635bc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1682560587295 2023-04-27 01:56:29,250 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 606b2c3ce38c4508824a43237c82d588, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682560588447 2023-04-27 01:56:29,255 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#C#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:29,270 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/bd3e097e729d47a5ae4e58077bb75c8e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bd3e097e729d47a5ae4e58077bb75c8e 2023-04-27 01:56:29,273 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/C of ec4ba6d393d7d8e622c952e9d8a3e627 into bd3e097e729d47a5ae4e58077bb75c8e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:29,274 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:29,274 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/C, priority=13, startTime=1682560589213; duration=0sec 2023-04-27 01:56:29,274 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:29,323 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:29,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:29,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:29,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:29,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:29,326 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:29,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:29,326 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:29,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,327 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,328 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:29,328 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,329 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,329 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:29,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:29,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:29,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:29,330 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:29,330 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:29,331 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 01:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:29,331 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:29,346 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304275a68cba194bb438aa2ffa79dc19931f8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275a68cba194bb438aa2ffa79dc19931f8 2023-04-27 01:56:29,358 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=339, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/ddd1b4213bde499a8ef9041f8c073123 2023-04-27 01:56:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:29,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:29,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:29,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560649440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:29,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560649543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,659 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/5d5a305b17d5450db96d01f05c36671b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5d5a305b17d5450db96d01f05c36671b 2023-04-27 01:56:29,663 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/A of ec4ba6d393d7d8e622c952e9d8a3e627 into 5d5a305b17d5450db96d01f05c36671b(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:29,663 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:29,663 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/A, priority=13, startTime=1682560589213; duration=0sec 2023-04-27 01:56:29,663 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:29,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:29,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560649746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,768 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/ea7c54555bd744a8ab311760b687289e 2023-04-27 01:56:29,778 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/96f009cca0194d57a3de44697d1a8d89 2023-04-27 01:56:29,782 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/ddd1b4213bde499a8ef9041f8c073123 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/ddd1b4213bde499a8ef9041f8c073123 2023-04-27 01:56:29,786 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/ddd1b4213bde499a8ef9041f8c073123, entries=150, sequenceid=339, filesize=25.4 K 2023-04-27 01:56:29,787 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/ea7c54555bd744a8ab311760b687289e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/ea7c54555bd744a8ab311760b687289e 2023-04-27 01:56:29,790 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/ea7c54555bd744a8ab311760b687289e, entries=150, sequenceid=339, filesize=11.9 K 2023-04-27 01:56:29,791 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/96f009cca0194d57a3de44697d1a8d89 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/96f009cca0194d57a3de44697d1a8d89 2023-04-27 01:56:29,794 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/96f009cca0194d57a3de44697d1a8d89, entries=150, sequenceid=339, filesize=11.9 K 2023-04-27 01:56:29,795 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ec4ba6d393d7d8e622c952e9d8a3e627 in 464ms, sequenceid=339, compaction requested=false 2023-04-27 01:56:29,795 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:29,795 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:29,796 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,796 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,797 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ba8aaa[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:29,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:29,798 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,804 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:29,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:29,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,804 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:29,804 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:29,805 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:29,809 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:29,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:29,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:29,809 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:29,809 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:29,810 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,810 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,810 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,810 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,812 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,816 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,816 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:29,816 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:29,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:29,816 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:29,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:29,816 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:29,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,817 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:29,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:29,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,817 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:29,817 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,917 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:29,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:29,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:29,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:29,921 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:29,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:29,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:29,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,921 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,923 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:29,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:29,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,923 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,924 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:29,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:29,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:29,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:29,925 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,925 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:29,925 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 01:56:29,925 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:29,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:29,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:29,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:29,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:29,926 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:29,940 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427d39bec6f57ca47bb91e1ae1acc7f7f8d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d39bec6f57ca47bb91e1ae1acc7f7f8d 2023-04-27 01:56:29,949 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/c28df876dee74898a256088fdd480160 2023-04-27 01:56:29,959 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/932011fe4ef94ff6aee18e279dea4332 2023-04-27 01:56:29,974 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/f9bd3cc52abb4889ac7a3524a55f30cd 2023-04-27 01:56:29,978 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/c28df876dee74898a256088fdd480160 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c28df876dee74898a256088fdd480160 2023-04-27 01:56:29,981 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c28df876dee74898a256088fdd480160, entries=150, sequenceid=356, filesize=25.4 K 2023-04-27 01:56:29,982 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/932011fe4ef94ff6aee18e279dea4332 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/932011fe4ef94ff6aee18e279dea4332 2023-04-27 01:56:29,985 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/932011fe4ef94ff6aee18e279dea4332, entries=150, sequenceid=356, filesize=11.9 K 2023-04-27 01:56:29,986 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/f9bd3cc52abb4889ac7a3524a55f30cd as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f9bd3cc52abb4889ac7a3524a55f30cd 2023-04-27 01:56:29,989 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f9bd3cc52abb4889ac7a3524a55f30cd, entries=150, sequenceid=356, filesize=11.9 K 2023-04-27 01:56:29,990 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for ec4ba6d393d7d8e622c952e9d8a3e627 in 64ms, sequenceid=356, compaction requested=true 2023-04-27 01:56:29,990 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:29,990 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:29,990 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:29,991 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50abfc77[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:29,994 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:29,994 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,995 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:29,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,995 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:29,996 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:29,999 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:29,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:29,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:29,999 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:29,999 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,001 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,001 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,001 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,002 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,003 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,006 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,006 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,005 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,005 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,106 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:30,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:30,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:30,109 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:30,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:30,109 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:30,110 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,110 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:30,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 01:56:30,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:30,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:30,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:30,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:30,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:30,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:30,111 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,112 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,112 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,112 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:30,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:30,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:30,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,114 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:30,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:30,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:30,120 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,123 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@421fce46[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:30,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:30,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,128 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:30,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:30,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:30,130 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:30,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:30,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:30,130 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:30,130 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,139 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,139 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:30,140 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,140 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,140 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427542e012d18744d93b26502d745e17a02 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427542e012d18744d93b26502d745e17a02 2023-04-27 01:56:30,150 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/51691a21c7cd45469dd83f7c81cf7626 2023-04-27 01:56:30,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/efc893b9a19345ce835873b22efa3719 2023-04-27 01:56:30,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/1e21c436851243c39992ae38e2d1c14b 2023-04-27 01:56:30,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/51691a21c7cd45469dd83f7c81cf7626 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/51691a21c7cd45469dd83f7c81cf7626 2023-04-27 01:56:30,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/51691a21c7cd45469dd83f7c81cf7626, entries=200, sequenceid=366, filesize=32.3 K 2023-04-27 01:56:30,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/efc893b9a19345ce835873b22efa3719 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/efc893b9a19345ce835873b22efa3719 2023-04-27 01:56:30,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/efc893b9a19345ce835873b22efa3719, entries=150, sequenceid=366, filesize=11.9 K 2023-04-27 01:56:30,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/1e21c436851243c39992ae38e2d1c14b as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1e21c436851243c39992ae38e2d1c14b 2023-04-27 01:56:30,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1e21c436851243c39992ae38e2d1c14b, entries=150, sequenceid=366, filesize=11.9 K 2023-04-27 01:56:30,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=93.93 KB/96180 for ec4ba6d393d7d8e622c952e9d8a3e627 in 79ms, sequenceid=366, compaction requested=true 2023-04-27 01:56:30,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:30,190 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:30,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:30,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:30,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:30,190 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:30,191 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111974 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:30,191 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/A is initiating minor compaction (all files) 2023-04-27 01:56:30,191 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/A in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,191 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:30,191 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/B is initiating minor compaction (all files) 2023-04-27 01:56:30,191 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/B in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,191 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/78a5b88772d34a5ebecf6c7d88410964, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/ea7c54555bd744a8ab311760b687289e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/932011fe4ef94ff6aee18e279dea4332, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/efc893b9a19345ce835873b22efa3719] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=48.3 K 2023-04-27 01:56:30,191 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5d5a305b17d5450db96d01f05c36671b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/ddd1b4213bde499a8ef9041f8c073123, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c28df876dee74898a256088fdd480160, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/51691a21c7cd45469dd83f7c81cf7626] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=109.3 K 2023-04-27 01:56:30,192 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 78a5b88772d34a5ebecf6c7d88410964, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682560588447 2023-04-27 01:56:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:30,192 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 5d5a305b17d5450db96d01f05c36671b, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682560588447 2023-04-27 01:56:30,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:56:30,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:30,193 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting ea7c54555bd744a8ab311760b687289e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1682560588760 2023-04-27 01:56:30,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:30,193 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting ddd1b4213bde499a8ef9041f8c073123, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1682560588760 2023-04-27 01:56:30,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:30,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:30,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:30,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:30,193 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 932011fe4ef94ff6aee18e279dea4332, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1682560589420 2023-04-27 01:56:30,194 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting c28df876dee74898a256088fdd480160, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1682560589420 2023-04-27 01:56:30,194 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting efc893b9a19345ce835873b22efa3719, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682560590103 2023-04-27 01:56:30,194 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 51691a21c7cd45469dd83f7c81cf7626, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682560590097 2023-04-27 01:56:30,207 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#B#compaction#578 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:30,223 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#A#compaction#577 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:30,228 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427196d5534d9c0470aaf35c522c26f0d20 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427196d5534d9c0470aaf35c522c26f0d20 2023-04-27 01:56:30,236 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/b17faa3652234988a96d5e8659a89bef as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b17faa3652234988a96d5e8659a89bef 2023-04-27 01:56:30,240 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:30,241 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/B of ec4ba6d393d7d8e622c952e9d8a3e627 into b17faa3652234988a96d5e8659a89bef(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:30,241 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:30,241 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/B, priority=12, startTime=1682560590190; duration=0sec 2023-04-27 01:56:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:30,241 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:30,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560650238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,241 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 01:56:30,243 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 01:56:30,243 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/C is initiating minor compaction (all files) 2023-04-27 01:56:30,244 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/C in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,244 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bd3e097e729d47a5ae4e58077bb75c8e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/96f009cca0194d57a3de44697d1a8d89, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f9bd3cc52abb4889ac7a3524a55f30cd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1e21c436851243c39992ae38e2d1c14b] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=48.3 K 2023-04-27 01:56:30,244 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting bd3e097e729d47a5ae4e58077bb75c8e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682560588447 2023-04-27 01:56:30,244 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 96f009cca0194d57a3de44697d1a8d89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1682560588760 2023-04-27 01:56:30,245 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting f9bd3cc52abb4889ac7a3524a55f30cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1682560589420 2023-04-27 01:56:30,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:30,245 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1e21c436851243c39992ae38e2d1c14b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682560590103 2023-04-27 01:56:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:30,250 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:30,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:30,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:30,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,253 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:30,254 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#C#compaction#579 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:30,255 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:30,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:30,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:30,257 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,258 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:30,260 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,260 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:30,260 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:30,260 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:30,263 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/75e850a44b6b421b911edf2d7d32af7f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/75e850a44b6b421b911edf2d7d32af7f 2023-04-27 01:56:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,265 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:30,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16bc28a6[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:30,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:30,267 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,268 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:30,268 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,268 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:30,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:30,268 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:30,268 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/A of ec4ba6d393d7d8e622c952e9d8a3e627 into 75e850a44b6b421b911edf2d7d32af7f(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:30,268 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:30,269 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/A, priority=12, startTime=1682560590190; duration=0sec 2023-04-27 01:56:30,269 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,270 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:30,270 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:30,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:30,272 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:30,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:30,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:30,272 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:30,272 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,277 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,280 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,281 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,283 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,283 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,283 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,284 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,283 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:30,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560650342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,384 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:30,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:30,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:30,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:30,387 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:30,387 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,387 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:30,387 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:30,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,389 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,390 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:30,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:30,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:30,391 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,391 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:30,391 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:30,391 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:30,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,393 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:30,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e71f01[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:30,395 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:30,395 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,398 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,398 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:30,398 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:30,398 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:30,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:30,399 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,399 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,400 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,400 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:30,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:30,403 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:30,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:30,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:30,403 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:30,403 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,404 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,405 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,406 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,408 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,408 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:30,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,409 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,408 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,409 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,409 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,509 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:30,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:30,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:30,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:30,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:30,513 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:30,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:30,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:30,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,513 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,515 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:30,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:30,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:30,517 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,517 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:30,517 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:30,517 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:30,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,519 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:30,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da0076e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:30,521 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:30,521 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,522 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,522 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,522 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:30,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:30,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:30,522 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:30,523 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,523 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:30,523 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,523 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:30,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:30,526 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:30,526 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:30,526 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:30,526 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:30,527 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,529 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,529 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,529 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,530 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,531 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,533 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,533 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,534 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,534 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560650545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,634 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:30,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:30,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:30,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:30,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:30,637 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:30,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:30,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:30,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,639 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,639 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,639 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,639 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:30,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:30,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:30,640 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:30,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:30,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:30,641 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:30,641 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:30,642 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,642 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,644 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=384, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/044d596049d9489794eb00605dd6cd48 2023-04-27 01:56:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11718162[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:30,645 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:30,645 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,646 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,647 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:30,647 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,647 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:30,647 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:30,647 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:30,648 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,649 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:30,649 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,649 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:30,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:30,653 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:30,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:30,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:30,653 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:30,654 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43550 deadline: 1682560650654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,656 DEBUG [Thread-3734] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:30,656 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,656 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,656 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,657 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,658 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43538 deadline: 1682560650658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,661 DEBUG [Thread-3732] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:30,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,664 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/5985b139f1944126b112b079e73a764f as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/5985b139f1944126b112b079e73a764f 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,666 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,667 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:30,667 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,667 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,667 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43558 deadline: 1682560650668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,670 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/C of ec4ba6d393d7d8e622c952e9d8a3e627 into 5985b139f1944126b112b079e73a764f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:30,670 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:30,670 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/C, priority=12, startTime=1682560590190; duration=0sec 2023-04-27 01:56:30,670 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:30,671 DEBUG [Thread-3728] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:30,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43526 deadline: 1682560650690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,693 DEBUG [Thread-3726] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., hostname=jenkins-hbase4.apache.org,38085,1682560417985, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 01:56:30,767 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:30,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:30,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:30,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:30,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:30,770 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:30,770 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:30,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:30,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,772 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,772 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,772 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,773 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:30,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:30,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:30,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:30,773 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:30,774 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,774 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:30,774 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:30,774 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:30,776 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@667f62f2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:30,778 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:30,778 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,779 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,779 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:30,779 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:30,779 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:30,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,780 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:30,782 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:30,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,782 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:30,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:30,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:30,785 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:30,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:30,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:30,785 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:30,785 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,786 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,787 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,788 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,790 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,790 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,791 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,791 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,791 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560650850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,891 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:30,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:30,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:30,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:30,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:30,894 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:30,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:30,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:30,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,895 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,896 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:30,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:30,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:30,897 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,897 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:30,897 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:30,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:30,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:30,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfa762b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:30,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:30,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:30,903 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,903 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:30,903 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:30,904 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:30,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:30,906 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:30,906 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:30,906 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:30,906 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:30,906 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:30,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,908 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:30,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:30,909 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,910 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:30,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:30,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:30,913 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:30,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:30,913 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:30,914 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:30,914 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:30,914 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:31,014 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:31,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:31,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:31,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:31,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:31,017 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:31,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:31,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:31,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:31,018 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,018 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:31,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:31,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:31,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:31,021 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:31,021 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:31,021 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:31,021 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:31,021 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:31,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cc95ad[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:31,025 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:31,025 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,026 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:31,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:31,026 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:31,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:31,028 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,028 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:31,028 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:31,028 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:31,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:31,031 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:31,031 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:31,031 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:31,031 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:31,031 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:31,033 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,033 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:31,033 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,033 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:31,035 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:31,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:31,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:31,040 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:31,040 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:31,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:31,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:31,040 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:31,040 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:31,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:31,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/8c94c2bafba84014b9601ec046b05164 2023-04-27 01:56:31,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/db1adb491ce64a4695a3272eb6bc9992 2023-04-27 01:56:31,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/044d596049d9489794eb00605dd6cd48 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/044d596049d9489794eb00605dd6cd48 2023-04-27 01:56:31,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/044d596049d9489794eb00605dd6cd48, entries=200, sequenceid=384, filesize=32.3 K 2023-04-27 01:56:31,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/8c94c2bafba84014b9601ec046b05164 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8c94c2bafba84014b9601ec046b05164 2023-04-27 01:56:31,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8c94c2bafba84014b9601ec046b05164, entries=150, sequenceid=384, filesize=11.9 K 2023-04-27 01:56:31,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/db1adb491ce64a4695a3272eb6bc9992 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/db1adb491ce64a4695a3272eb6bc9992 2023-04-27 01:56:31,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/db1adb491ce64a4695a3272eb6bc9992, entries=150, sequenceid=384, filesize=11.9 K 2023-04-27 01:56:31,077 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ec4ba6d393d7d8e622c952e9d8a3e627 in 885ms, sequenceid=384, compaction requested=false 2023-04-27 01:56:31,077 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:31,140 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:31,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:31,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:31,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:31,144 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:31,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:31,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:31,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:31,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,144 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,146 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:31,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:31,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,146 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:31,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:31,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:31,148 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:31,148 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:31,164 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e2e560428215450c9d8efdbee0ce30a0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e2e560428215450c9d8efdbee0ce30a0 2023-04-27 01:56:31,169 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/e9e949afcfea4d74921404a23587c6b0 2023-04-27 01:56:31,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:31,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:31,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:31,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560651395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:31,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560651497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,586 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/bac67d59b85d486899daeec7dbc3b57a 2023-04-27 01:56:31,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:31,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38085] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560651701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:31,997 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/4da956f473ee4d189326135aab4d3165 2023-04-27 01:56:32,000 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/e9e949afcfea4d74921404a23587c6b0 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e9e949afcfea4d74921404a23587c6b0 2023-04-27 01:56:32,003 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e9e949afcfea4d74921404a23587c6b0, entries=150, sequenceid=401, filesize=25.4 K 2023-04-27 01:56:32,004 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/bac67d59b85d486899daeec7dbc3b57a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/bac67d59b85d486899daeec7dbc3b57a 2023-04-27 01:56:32,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:32,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560652003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,007 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/bac67d59b85d486899daeec7dbc3b57a, entries=150, sequenceid=401, filesize=11.9 K 2023-04-27 01:56:32,007 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/4da956f473ee4d189326135aab4d3165 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/4da956f473ee4d189326135aab4d3165 2023-04-27 01:56:32,013 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/4da956f473ee4d189326135aab4d3165, entries=150, sequenceid=401, filesize=11.9 K 2023-04-27 01:56:32,013 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ec4ba6d393d7d8e622c952e9d8a3e627 in 865ms, sequenceid=401, compaction requested=true 2023-04-27 01:56:32,014 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:32,014 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:32,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:32,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:32,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:32,014 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:32,015 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,015 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,016 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:32,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c2b6467[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:32,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:32,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,018 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,018 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:32,018 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:32,018 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:32,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:32,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,019 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,020 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:32,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:32,022 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:32,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:32,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:32,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:32,022 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:32,025 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,025 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,025 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,025 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,027 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,029 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:32,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:32,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,029 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,029 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:32,029 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:32,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:32,130 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:32,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:32,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:32,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:32,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:32,133 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:32,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:32,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:32,133 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:32,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,135 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:32,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:32,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:32,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:32,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:32,137 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:32,137 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:32,137 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 01:56:32,137 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:32,137 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:32,138 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:32,138 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:32,138 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:32,138 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:32,157 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304278fc21305be1e483597671ca35ec3528d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278fc21305be1e483597671ca35ec3528d 2023-04-27 01:56:32,165 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/8626a44a8271457d9d6ebdba07e3a7b8 2023-04-27 01:56:32,185 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/950e050916b145a7b9b48fa002a18629 2023-04-27 01:56:32,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:32,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:32,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560652540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,596 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3c946e12a8874c318fd0d3ee4befcb26 2023-04-27 01:56:32,601 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/8626a44a8271457d9d6ebdba07e3a7b8 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/8626a44a8271457d9d6ebdba07e3a7b8 2023-04-27 01:56:32,604 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/8626a44a8271457d9d6ebdba07e3a7b8, entries=150, sequenceid=419, filesize=25.4 K 2023-04-27 01:56:32,606 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/950e050916b145a7b9b48fa002a18629 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/950e050916b145a7b9b48fa002a18629 2023-04-27 01:56:32,617 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/950e050916b145a7b9b48fa002a18629, entries=150, sequenceid=419, filesize=11.9 K 2023-04-27 01:56:32,618 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/3c946e12a8874c318fd0d3ee4befcb26 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3c946e12a8874c318fd0d3ee4befcb26 2023-04-27 01:56:32,622 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3c946e12a8874c318fd0d3ee4befcb26, entries=150, sequenceid=419, filesize=11.9 K 2023-04-27 01:56:32,622 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ec4ba6d393d7d8e622c952e9d8a3e627 in 485ms, sequenceid=419, compaction requested=true 2023-04-27 01:56:32,623 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:32,623 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,625 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:32,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515ef7e2[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:32,626 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:32,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,633 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,633 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,633 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:32,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:32,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:32,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:32,634 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:32,635 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,635 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,635 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:32,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:32,637 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:32,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:32,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:32,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:32,637 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:32,639 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,639 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,639 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,639 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,641 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:32,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:32,643 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,644 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,644 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:32,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:32,644 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,644 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:32,644 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:32,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(9158): Flush requested on ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:32,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 01:56:32,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:32,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:32,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:32,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:32,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:32,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:32,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:32,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560652693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,744 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:32,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:32,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:32,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:32,747 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:32,747 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,748 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,750 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:32,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:32,750 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,750 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,750 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:32,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:32,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:32,752 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:32,752 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:32,752 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:32,752 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:32,753 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:32,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eba98cc[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:32,755 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:32,755 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,756 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,756 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:32,756 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:32,756 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:32,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:32,757 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,758 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:32,758 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,758 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:32,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:32,760 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:32,760 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:32,760 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:32,760 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:32,760 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:32,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,767 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,770 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:32,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:32,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,770 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,770 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:32,770 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:32,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560652796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,870 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:32,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:32,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:32,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:32,874 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:32,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:32,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:32,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:32,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,874 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,875 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:32,875 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:32,876 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,876 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:32,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:32,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:32,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:32,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:32,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:32,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:32,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:32,877 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:32,877 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:32,877 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:32,877 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:32,877 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:32,878 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:32,878 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,878 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:32,878 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:32,887 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,887 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,887 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,887 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:32,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6e1b4b[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:32,889 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:32,889 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,890 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,890 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,890 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:32,890 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:32,890 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:32,890 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:32,891 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:32,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:32,894 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:32,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:32,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:32,894 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:32,894 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:32,895 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,895 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,895 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:32,896 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,897 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:32,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:32,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:32,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:32,900 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:32,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:32,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:32,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:32,900 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,901 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:32,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:32,901 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:32,901 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:32,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:32,901 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:33,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560653000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,001 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,004 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,004 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,006 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,006 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,006 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,006 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,007 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,007 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,007 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,007 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,008 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,009 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,009 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,009 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,010 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c71461d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,011 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,011 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,012 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,012 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,015 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,015 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,017 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,017 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,017 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,019 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,019 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,020 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,022 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,022 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,022 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,022 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:33,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,023 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,023 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,023 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,061 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304270e2d243c22ac4bb99ce5138457b3ed2b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270e2d243c22ac4bb99ce5138457b3ed2b 2023-04-27 01:56:33,123 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,126 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,126 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,127 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,128 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,128 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,129 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,129 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,130 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,130 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,130 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,130 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,130 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,132 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,132 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,133 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,133 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a364b96[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,134 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,135 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,135 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,135 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,136 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,136 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,136 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,136 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,138 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,138 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,139 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,139 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,139 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,140 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,140 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,140 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,141 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,142 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,145 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,145 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:33,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,145 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,146 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,246 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,249 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,249 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,250 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,250 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,251 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,251 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,252 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,252 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,252 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,253 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,253 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,253 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,253 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,253 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,253 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,253 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,254 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,254 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5060d0e4[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,256 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,256 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,264 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,264 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,264 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,264 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,264 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,264 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,265 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,265 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,266 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,266 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,268 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,269 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,269 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,270 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,270 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,270 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,271 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,273 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,275 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:33,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,275 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,275 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,275 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:33,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38085] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560653301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,376 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,379 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,379 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,379 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,379 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,380 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,380 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,382 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,382 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,383 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,383 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,384 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,384 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,385 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,386 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bef3de0[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,388 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,389 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,389 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,389 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,389 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,389 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,390 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,390 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,391 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,393 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,393 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,393 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,396 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,396 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,396 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,397 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,398 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,401 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,401 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,402 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,402 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,467 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=434, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/f707c471e8064d299729e1da9816f6af 2023-04-27 01:56:33,502 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,505 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,505 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,505 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,505 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,506 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,506 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,507 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,509 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,509 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,509 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,509 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,511 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,511 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,511 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,511 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d78cf2e[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,512 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,515 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,515 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,515 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,515 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,515 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,516 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,517 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,517 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,519 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,519 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,519 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,519 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,519 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,521 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,521 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,521 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,522 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,523 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:33,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,525 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,525 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,525 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,526 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,526 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,526 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,626 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,629 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,629 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,631 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,631 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,632 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,632 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,633 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,633 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,633 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,633 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,633 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,634 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,634 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,634 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,635 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755a59ec[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,636 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,637 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,638 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,638 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,643 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,643 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,646 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,646 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,646 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,646 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,646 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,647 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,647 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,647 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,648 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,649 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,652 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,652 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,653 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:33,653 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,753 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,756 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,756 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,757 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,758 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,759 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,759 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,760 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,760 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,760 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,760 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,760 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,763 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@236b4053[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,765 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,765 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,766 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,766 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,766 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,766 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,766 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,768 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,768 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,771 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,771 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,771 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,772 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,772 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,772 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,773 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,774 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,777 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,777 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:33,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,778 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,778 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 01:56:33,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38085] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43496 deadline: 1682560653804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/1ccc4d9acea34cf1adae2c7c7b42930a 2023-04-27 01:56:33,878 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:33,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:33,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:33,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:33,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:33,884 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:33,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:33,884 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:33,885 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,885 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,886 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,886 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,887 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,887 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:33,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:33,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:33,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:33,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:33,896 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:33,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:33,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. as already flushing 2023-04-27 01:56:33,898 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:33,899 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,900 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/6df82101450949cfa2faae47a987a091 2023-04-27 01:56:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f969e3[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:33,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:33,902 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,903 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:33,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:33,904 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:33,904 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,904 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:33,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/f707c471e8064d299729e1da9816f6af as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/f707c471e8064d299729e1da9816f6af 2023-04-27 01:56:33,906 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,906 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:33,907 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,907 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/f707c471e8064d299729e1da9816f6af, entries=150, sequenceid=434, filesize=25.4 K 2023-04-27 01:56:33,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/1ccc4d9acea34cf1adae2c7c7b42930a as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/1ccc4d9acea34cf1adae2c7c7b42930a 2023-04-27 01:56:33,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:33,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:33,911 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:33,911 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:33,911 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:33,911 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:33,911 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:33,912 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,912 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:33,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,912 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:33,913 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/1ccc4d9acea34cf1adae2c7c7b42930a, entries=150, sequenceid=434, filesize=11.9 K 2023-04-27 01:56:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:33,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/6df82101450949cfa2faae47a987a091 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6df82101450949cfa2faae47a987a091 2023-04-27 01:56:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,915 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,917 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:33,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:33,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:33,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6df82101450949cfa2faae47a987a091, entries=150, sequenceid=434, filesize=11.9 K 2023-04-27 01:56:33,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:33,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:33,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:33,918 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:33,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:33,918 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:33,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:33,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:33,918 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ec4ba6d393d7d8e622c952e9d8a3e627 in 1272ms, sequenceid=434, compaction requested=true 2023-04-27 01:56:33,919 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:33,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:33,919 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:33,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:33,919 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:33,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:33,919 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:33,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:33,920 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138156 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:33,920 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/A is initiating minor compaction (all files) 2023-04-27 01:56:33,920 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/A in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,920 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/75e850a44b6b421b911edf2d7d32af7f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/044d596049d9489794eb00605dd6cd48, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e9e949afcfea4d74921404a23587c6b0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/8626a44a8271457d9d6ebdba07e3a7b8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/f707c471e8064d299729e1da9816f6af] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=134.9 K 2023-04-27 01:56:33,921 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:33,921 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/B is initiating minor compaction (all files) 2023-04-27 01:56:33,921 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 75e850a44b6b421b911edf2d7d32af7f, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682560590103 2023-04-27 01:56:33,921 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/B in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,921 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b17faa3652234988a96d5e8659a89bef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8c94c2bafba84014b9601ec046b05164, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/bac67d59b85d486899daeec7dbc3b57a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/950e050916b145a7b9b48fa002a18629, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/1ccc4d9acea34cf1adae2c7c7b42930a] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=60.4 K 2023-04-27 01:56:33,921 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 044d596049d9489794eb00605dd6cd48, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682560590156 2023-04-27 01:56:33,921 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting b17faa3652234988a96d5e8659a89bef, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682560590103 2023-04-27 01:56:33,921 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting e9e949afcfea4d74921404a23587c6b0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682560590221 2023-04-27 01:56:33,922 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 8c94c2bafba84014b9601ec046b05164, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682560590156 2023-04-27 01:56:33,922 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting 8626a44a8271457d9d6ebdba07e3a7b8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682560591386 2023-04-27 01:56:33,922 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting bac67d59b85d486899daeec7dbc3b57a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682560590221 2023-04-27 01:56:33,922 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 950e050916b145a7b9b48fa002a18629, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682560591386 2023-04-27 01:56:33,922 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] compactions.Compactor(207): Compacting f707c471e8064d299729e1da9816f6af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682560592532 2023-04-27 01:56:33,923 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 1ccc4d9acea34cf1adae2c7c7b42930a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682560592532 2023-04-27 01:56:33,932 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#B#compaction#592 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:33,937 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#A#compaction#591 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:33,951 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/34132b65b9124423a599eafa88b5a43e as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/34132b65b9124423a599eafa88b5a43e 2023-04-27 01:56:33,955 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/B of ec4ba6d393d7d8e622c952e9d8a3e627 into 34132b65b9124423a599eafa88b5a43e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:33,955 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:33,955 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/B, priority=11, startTime=1682560593919; duration=0sec 2023-04-27 01:56:33,956 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 01:56:33,956 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 01:56:33,957 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 01:56:33,957 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1912): ec4ba6d393d7d8e622c952e9d8a3e627/C is initiating minor compaction (all files) 2023-04-27 01:56:33,957 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ec4ba6d393d7d8e622c952e9d8a3e627/C in TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:33,957 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/5985b139f1944126b112b079e73a764f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/db1adb491ce64a4695a3272eb6bc9992, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/4da956f473ee4d189326135aab4d3165, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3c946e12a8874c318fd0d3ee4befcb26, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6df82101450949cfa2faae47a987a091] into tmpdir=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp, totalSize=60.4 K 2023-04-27 01:56:33,958 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 5985b139f1944126b112b079e73a764f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682560590103 2023-04-27 01:56:33,958 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting db1adb491ce64a4695a3272eb6bc9992, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682560590156 2023-04-27 01:56:33,958 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 4da956f473ee4d189326135aab4d3165, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682560590221 2023-04-27 01:56:33,959 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 3c946e12a8874c318fd0d3ee4befcb26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682560591386 2023-04-27 01:56:33,959 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] compactions.Compactor(207): Compacting 6df82101450949cfa2faae47a987a091, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682560592532 2023-04-27 01:56:33,967 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] throttle.PressureAwareThroughputController(145): ec4ba6d393d7d8e622c952e9d8a3e627#C#compaction#593 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 01:56:33,976 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/490592d07b914524999392853a29b340 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/490592d07b914524999392853a29b340 2023-04-27 01:56:33,981 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/C of ec4ba6d393d7d8e622c952e9d8a3e627 into 490592d07b914524999392853a29b340(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:33,981 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:33,981 INFO [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/C, priority=11, startTime=1682560593919; duration=0sec 2023-04-27 01:56:33,981 DEBUG [RS:0;jenkins-hbase4:38085-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:34,019 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:34,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:34,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:34,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:34,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:34,024 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:34,024 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:34,024 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:34,024 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:34,024 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,024 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,025 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:34,025 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:34,026 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,026 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:34,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:34,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:34,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:34,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:34,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:34,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:34,027 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:34,027 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:34,027 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:34,027 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 01:56:34,028 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:34,028 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:34,028 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:34,028 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:34,028 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:34,028 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:34,372 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/5fe4fe2b0ec349e391a8ca4aa4b1eda5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5fe4fe2b0ec349e391a8ca4aa4b1eda5 2023-04-27 01:56:34,376 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ec4ba6d393d7d8e622c952e9d8a3e627/A of ec4ba6d393d7d8e622c952e9d8a3e627 into 5fe4fe2b0ec349e391a8ca4aa4b1eda5(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 01:56:34,376 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:34,376 INFO [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627., storeName=ec4ba6d393d7d8e622c952e9d8a3e627/A, priority=11, startTime=1682560593919; duration=0sec 2023-04-27 01:56:34,376 DEBUG [RS:0;jenkins-hbase4:38085-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 01:56:34,443 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427f3c03bd14c3d4a2dbcaa44c812a3c103 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f3c03bd14c3d4a2dbcaa44c812a3c103 2023-04-27 01:56:34,449 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=453, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/b5e95649448e4c0e82373629658c4f56 2023-04-27 01:56:34,462 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/92192bd39da34539b76187133acd7c42 2023-04-27 01:56:34,470 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/674e5e488c57402e9a62c29f0d539f24 2023-04-27 01:56:34,475 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/b5e95649448e4c0e82373629658c4f56 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b5e95649448e4c0e82373629658c4f56 2023-04-27 01:56:34,479 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b5e95649448e4c0e82373629658c4f56, entries=150, sequenceid=453, filesize=25.4 K 2023-04-27 01:56:34,479 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/92192bd39da34539b76187133acd7c42 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/92192bd39da34539b76187133acd7c42 2023-04-27 01:56:34,482 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/92192bd39da34539b76187133acd7c42, entries=150, sequenceid=453, filesize=11.9 K 2023-04-27 01:56:34,483 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/674e5e488c57402e9a62c29f0d539f24 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/674e5e488c57402e9a62c29f0d539f24 2023-04-27 01:56:34,489 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/674e5e488c57402e9a62c29f0d539f24, entries=150, sequenceid=453, filesize=11.9 K 2023-04-27 01:56:34,490 INFO [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for ec4ba6d393d7d8e622c952e9d8a3e627 in 463ms, sequenceid=453, compaction requested=false 2023-04-27 01:56:34,490 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:34,490 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:34,492 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a03d14d[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:34,494 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:34,494 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,495 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,495 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:34,495 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:34,495 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,495 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:34,495 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:34,496 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:34,497 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:34,497 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,497 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:34,502 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:34,502 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:34,502 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:34,502 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:34,502 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:34,505 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,505 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,505 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:34,506 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,507 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,511 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,511 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:34,511 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:34,511 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,511 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,511 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:34,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:34,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:34,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:34,511 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,512 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,512 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,512 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,512 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:34,512 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:34,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:34,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:34,612 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:34,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:34,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:34,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:34,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:34,616 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:34,616 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:34,616 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:34,616 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:34,616 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,616 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,620 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,620 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:34,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:34,620 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,621 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:34,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:34,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:34,622 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:34,622 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:34,622 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:34,622 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,623 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:34,623 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:34,625 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,626 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,626 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:34,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3164d8[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:34,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:34,627 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,628 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:34,628 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,628 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:34,629 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:34,630 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:34,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,630 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:34,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:34,632 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:34,633 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:34,633 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:34,633 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:34,633 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:34,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:34,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,634 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,635 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,636 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,640 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:34,640 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:34,640 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:34,640 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:34,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:34,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:34,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:34,640 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,641 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,641 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,641 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,641 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,641 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,641 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:34,641 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:34,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:34,741 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:34,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 01:56:34,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 01:56:34,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 01:56:34,744 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 01:56:34,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:34,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 01:56:34,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 01:56:34,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,745 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,746 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,746 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:34,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,747 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,747 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 01:56:34,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 01:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 01:56:34,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 01:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 01:56:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 01:56:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 01:56:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 01:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 01:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 01:56:34,749 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:34,749 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. started... 2023-04-27 01:56:34,749 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:34,749 DEBUG [rs(jenkins-hbase4.apache.org,38085,1682560417985)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 01:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 01:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 01:56:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 01:56:34,750 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,751 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 01:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38085,1682560417985' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 01:56:34,752 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 01:56:34,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a1d8f5[Count = 0] remaining members to acquire global barrier 2023-04-27 01:56:34,752 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,754 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,754 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 01:56:34,754 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,754 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 01:56:34,754 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 01:56:34,754 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38085,1682560417985' in zk 2023-04-27 01:56:34,756 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 01:56:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:34,756 DEBUG [member: 'jenkins-hbase4.apache.org,38085,1682560417985' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 01:56:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38085,1682560417985': 2023-04-27 01:56:34,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38085,1682560417985' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 01:56:34,758 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 01:56:34,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 01:56:34,758 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 01:56:34,759 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 01:56:34,759 INFO [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 01:56:34,760 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,760 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,760 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 01:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 01:56:34,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:34,761 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 01:56:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 01:56:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,762 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 01:56:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 01:56:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,764 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 01:56:34,764 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 01:56:34,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 01:56:34,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 01:56:34,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 01:56:34,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 01:56:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 01:56:34,764 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 01:56:34,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:34,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 01:56:34,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 01:56:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 01:56:34,764 DEBUG [(jenkins-hbase4.apache.org,34643,1682560416787)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 01:56:34,765 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 01:56:34,765 DEBUG [Thread-3736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 01:56:34,765 DEBUG [Thread-3736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 01:56:34,787 DEBUG [Thread-3745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f8e4d5e to 127.0.0.1:52731 2023-04-27 01:56:34,787 DEBUG [Thread-3745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:34,787 DEBUG [Thread-3737] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17cf2e3f to 127.0.0.1:52731 2023-04-27 01:56:34,787 DEBUG [Thread-3737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:34,790 DEBUG [Thread-3743] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x601410a4 to 127.0.0.1:52731 2023-04-27 01:56:34,790 DEBUG [Thread-3743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:34,803 DEBUG [Thread-3741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x26d7d0b5 to 127.0.0.1:52731 2023-04-27 01:56:34,804 DEBUG [Thread-3741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:34,804 DEBUG [Thread-3739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3420e54c to 127.0.0.1:52731 2023-04-27 01:56:34,804 DEBUG [Thread-3739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:34,814 DEBUG [Thread-3730] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5bcc330a to 127.0.0.1:52731 2023-04-27 01:56:34,814 DEBUG [Thread-3730] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:34,865 DEBUG [Thread-3736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 01:56:34,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 01:56:40,690 DEBUG [Thread-3728] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01d207bb to 127.0.0.1:52731 2023-04-27 01:56:40,690 DEBUG [Thread-3728] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:40,718 DEBUG [Thread-3734] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f1adb60 to 127.0.0.1:52731 2023-04-27 01:56:40,718 DEBUG [Thread-3734] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:40,754 DEBUG [Thread-3726] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x089b6154 to 127.0.0.1:52731 2023-04-27 01:56:40,754 DEBUG [Thread-3726] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:40,755 DEBUG [Thread-3732] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23afc0f2 to 127.0.0.1:52731 2023-04-27 01:56:40,755 DEBUG [Thread-3732] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 217 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5096 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5030 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5076 2023-04-27 01:56:40,755 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5107 2023-04-27 01:56:40,756 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(398): read 5018 2023-04-27 01:56:40,756 INFO [Listener at localhost/42495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 01:56:40,756 INFO [Listener at localhost/42495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 01:56:40,756 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x041dc777 to 127.0.0.1:52731 2023-04-27 01:56:40,756 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:40,756 INFO [Listener at localhost/42495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 01:56:40,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 01:56:40,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:40,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 01:56:40,759 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560600759"}]},"ts":"1682560600759"} 2023-04-27 01:56:40,760 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 01:56:40,761 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 01:56:40,762 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, UNASSIGN}] 2023-04-27 01:56:40,763 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, UNASSIGN 2023-04-27 01:56:40,764 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:40,764 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560600763"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682560600763"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682560600763"}]},"ts":"1682560600763"} 2023-04-27 01:56:40,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985}] 2023-04-27 01:56:40,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 01:56:40,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:40,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ec4ba6d393d7d8e622c952e9d8a3e627, disabling compactions & flushes 2023-04-27 01:56:40,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:40,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:40,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. after waiting 0 ms 2023-04-27 01:56:40,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:40,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ec4ba6d393d7d8e622c952e9d8a3e627 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-27 01:56:40,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=A 2023-04-27 01:56:40,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:40,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=B 2023-04-27 01:56:40,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:40,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ec4ba6d393d7d8e622c952e9d8a3e627, store=C 2023-04-27 01:56:40,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 01:56:41,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 01:56:41,327 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304278a45df1817964ba89c90c30da4ac35d6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278a45df1817964ba89c90c30da4ac35d6 2023-04-27 01:56:41,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 01:56:41,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=464, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/49852e6d7e8e4f36b3e1b352540314b6 2023-04-27 01:56:41,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 01:56:42,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/8acc055c99c24953aba99029bbf4ff28 2023-04-27 01:56:42,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/847fa24031fd4a6f860ccb279609e617 2023-04-27 01:56:42,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/A/49852e6d7e8e4f36b3e1b352540314b6 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/49852e6d7e8e4f36b3e1b352540314b6 2023-04-27 01:56:42,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/49852e6d7e8e4f36b3e1b352540314b6, entries=150, sequenceid=464, filesize=25.4 K 2023-04-27 01:56:42,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/B/8acc055c99c24953aba99029bbf4ff28 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8acc055c99c24953aba99029bbf4ff28 2023-04-27 01:56:42,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8acc055c99c24953aba99029bbf4ff28, entries=150, sequenceid=464, filesize=11.9 K 2023-04-27 01:56:42,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/.tmp/C/847fa24031fd4a6f860ccb279609e617 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/847fa24031fd4a6f860ccb279609e617 2023-04-27 01:56:42,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/847fa24031fd4a6f860ccb279609e617, entries=150, sequenceid=464, filesize=11.9 K 2023-04-27 01:56:42,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for ec4ba6d393d7d8e622c952e9d8a3e627 in 1646ms, sequenceid=464, compaction requested=true 2023-04-27 01:56:42,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4c1b49df54804349b654f3cf3e83122a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/1a5fc6286e6b472193b082686f5b893e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b46aa787e8f84a97ab6dfe619d62b33b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d91e3bbbdf5146548f4de8a635d091e4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c3707fe8ddd543be942337739b1c7ed5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/9cf457438a054149a29e9f9719c3a46b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/de021e00e84c4301a10cf5a53ab74c14, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/48762217f78b4ee2930397a107f09e47, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/08f78cf8d1da4653beebdc287a91efa3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7469efaa34464e168babea6a03f5e1d5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7660e69c52194df7885a0bfbafb82081, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/811818fbf9074aed8439491fdf6bcca8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d39e58ba597b4c3ba4b44953e97c9af0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/93b9fcd31ecb4920aad1033d7ccf0a3a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/595303c23a80426b89dfb16cc3a4d827, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/00cc855220c24fba9529b80e2ce44d92, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/36a93504731145a7953a328636071ed2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4ef60a7db79546e7b0301c65a42e0d90, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/cdb7b428b39e4da4a16b4135ae847ded, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e683e5be33784b28b00cee320114921e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/27313dc531c24da29357fcd8c1e1f231, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b830f719a30f4a8b9572a8fc90c81b3c, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5d5a305b17d5450db96d01f05c36671b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/ddd1b4213bde499a8ef9041f8c073123, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c28df876dee74898a256088fdd480160, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/51691a21c7cd45469dd83f7c81cf7626, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/75e850a44b6b421b911edf2d7d32af7f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/044d596049d9489794eb00605dd6cd48, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e9e949afcfea4d74921404a23587c6b0, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/8626a44a8271457d9d6ebdba07e3a7b8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/f707c471e8064d299729e1da9816f6af] to archive 2023-04-27 01:56:42,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:56:42,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4c1b49df54804349b654f3cf3e83122a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4c1b49df54804349b654f3cf3e83122a 2023-04-27 01:56:42,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/1a5fc6286e6b472193b082686f5b893e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/1a5fc6286e6b472193b082686f5b893e 2023-04-27 01:56:42,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b46aa787e8f84a97ab6dfe619d62b33b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b46aa787e8f84a97ab6dfe619d62b33b 2023-04-27 01:56:42,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d91e3bbbdf5146548f4de8a635d091e4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d91e3bbbdf5146548f4de8a635d091e4 2023-04-27 01:56:42,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c3707fe8ddd543be942337739b1c7ed5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c3707fe8ddd543be942337739b1c7ed5 2023-04-27 01:56:42,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/9cf457438a054149a29e9f9719c3a46b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/9cf457438a054149a29e9f9719c3a46b 2023-04-27 01:56:42,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/de021e00e84c4301a10cf5a53ab74c14 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/de021e00e84c4301a10cf5a53ab74c14 2023-04-27 01:56:42,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/48762217f78b4ee2930397a107f09e47 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/48762217f78b4ee2930397a107f09e47 2023-04-27 01:56:42,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/08f78cf8d1da4653beebdc287a91efa3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/08f78cf8d1da4653beebdc287a91efa3 2023-04-27 01:56:42,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7469efaa34464e168babea6a03f5e1d5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7469efaa34464e168babea6a03f5e1d5 2023-04-27 01:56:42,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7660e69c52194df7885a0bfbafb82081 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/7660e69c52194df7885a0bfbafb82081 2023-04-27 01:56:42,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/811818fbf9074aed8439491fdf6bcca8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/811818fbf9074aed8439491fdf6bcca8 2023-04-27 01:56:42,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d39e58ba597b4c3ba4b44953e97c9af0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/d39e58ba597b4c3ba4b44953e97c9af0 2023-04-27 01:56:42,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/93b9fcd31ecb4920aad1033d7ccf0a3a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/93b9fcd31ecb4920aad1033d7ccf0a3a 2023-04-27 01:56:42,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/595303c23a80426b89dfb16cc3a4d827 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/595303c23a80426b89dfb16cc3a4d827 2023-04-27 01:56:42,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/00cc855220c24fba9529b80e2ce44d92 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/00cc855220c24fba9529b80e2ce44d92 2023-04-27 01:56:42,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/36a93504731145a7953a328636071ed2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/36a93504731145a7953a328636071ed2 2023-04-27 01:56:42,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4ef60a7db79546e7b0301c65a42e0d90 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/4ef60a7db79546e7b0301c65a42e0d90 2023-04-27 01:56:42,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/cdb7b428b39e4da4a16b4135ae847ded to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/cdb7b428b39e4da4a16b4135ae847ded 2023-04-27 01:56:42,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e683e5be33784b28b00cee320114921e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e683e5be33784b28b00cee320114921e 2023-04-27 01:56:42,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/27313dc531c24da29357fcd8c1e1f231 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/27313dc531c24da29357fcd8c1e1f231 2023-04-27 01:56:42,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b830f719a30f4a8b9572a8fc90c81b3c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b830f719a30f4a8b9572a8fc90c81b3c 2023-04-27 01:56:42,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5d5a305b17d5450db96d01f05c36671b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5d5a305b17d5450db96d01f05c36671b 2023-04-27 01:56:42,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/ddd1b4213bde499a8ef9041f8c073123 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/ddd1b4213bde499a8ef9041f8c073123 2023-04-27 01:56:42,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c28df876dee74898a256088fdd480160 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/c28df876dee74898a256088fdd480160 2023-04-27 01:56:42,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/51691a21c7cd45469dd83f7c81cf7626 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/51691a21c7cd45469dd83f7c81cf7626 2023-04-27 01:56:42,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/75e850a44b6b421b911edf2d7d32af7f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/75e850a44b6b421b911edf2d7d32af7f 2023-04-27 01:56:42,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/044d596049d9489794eb00605dd6cd48 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/044d596049d9489794eb00605dd6cd48 2023-04-27 01:56:42,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e9e949afcfea4d74921404a23587c6b0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/e9e949afcfea4d74921404a23587c6b0 2023-04-27 01:56:42,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/8626a44a8271457d9d6ebdba07e3a7b8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/8626a44a8271457d9d6ebdba07e3a7b8 2023-04-27 01:56:42,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/f707c471e8064d299729e1da9816f6af to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/f707c471e8064d299729e1da9816f6af 2023-04-27 01:56:42,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/128f8c843e8f4db38075ab05a03f2a98, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/098edf3a5f6248c497f02edba3c59f46, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/0d356718f7e241d0ba356e2bd5a2d77f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11ded7c218b406a90d15d98c1aae817, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/94cf8c6aa72949f5a1ee4ac98a4bcf7d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/17c15ece270d46faba40404e0393f7bd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3dcbf3f4cea24145a45360dcd0c271b3, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/4fb1e27e6e7e46a791cc7a41a2783012, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9b80c85f2e134048af6c5698ab716e87, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/197957d464394a9280f3a6e640874590, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/994be0785ebb492a95508800e1e6850f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/843da33b29654b998fb39c4cff712c25, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9bf8a1bd81044bd095570c13b2325462, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3388cfaea35743dd86e1d20718a9f9b2, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/7cc6d465fced49e8aff4a655c25ea5cb, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/877c33927f4148cb8d94b2ea63d1daee, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/d75c97259cc64e018f704214056c75b8, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/86b7ad9c6a2744ada258b34ceddebfd6, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9f32e4e2bc9f49f0a6e484677c724ae4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b18c3246154343059c353e3412ade4da, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/c8ea7b13dc8241229374968a3e6bd5f4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/78a5b88772d34a5ebecf6c7d88410964, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11f4003e42a4458a4948c9c63489e9d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/ea7c54555bd744a8ab311760b687289e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/932011fe4ef94ff6aee18e279dea4332, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b17faa3652234988a96d5e8659a89bef, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/efc893b9a19345ce835873b22efa3719, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8c94c2bafba84014b9601ec046b05164, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/bac67d59b85d486899daeec7dbc3b57a, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/950e050916b145a7b9b48fa002a18629, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/1ccc4d9acea34cf1adae2c7c7b42930a] to archive 2023-04-27 01:56:42,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:56:42,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/128f8c843e8f4db38075ab05a03f2a98 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/128f8c843e8f4db38075ab05a03f2a98 2023-04-27 01:56:42,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/098edf3a5f6248c497f02edba3c59f46 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/098edf3a5f6248c497f02edba3c59f46 2023-04-27 01:56:42,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/0d356718f7e241d0ba356e2bd5a2d77f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/0d356718f7e241d0ba356e2bd5a2d77f 2023-04-27 01:56:42,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11ded7c218b406a90d15d98c1aae817 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11ded7c218b406a90d15d98c1aae817 2023-04-27 01:56:42,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/94cf8c6aa72949f5a1ee4ac98a4bcf7d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/94cf8c6aa72949f5a1ee4ac98a4bcf7d 2023-04-27 01:56:42,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/17c15ece270d46faba40404e0393f7bd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/17c15ece270d46faba40404e0393f7bd 2023-04-27 01:56:42,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3dcbf3f4cea24145a45360dcd0c271b3 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3dcbf3f4cea24145a45360dcd0c271b3 2023-04-27 01:56:42,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/4fb1e27e6e7e46a791cc7a41a2783012 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/4fb1e27e6e7e46a791cc7a41a2783012 2023-04-27 01:56:42,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9b80c85f2e134048af6c5698ab716e87 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9b80c85f2e134048af6c5698ab716e87 2023-04-27 01:56:42,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/197957d464394a9280f3a6e640874590 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/197957d464394a9280f3a6e640874590 2023-04-27 01:56:42,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/994be0785ebb492a95508800e1e6850f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/994be0785ebb492a95508800e1e6850f 2023-04-27 01:56:42,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/843da33b29654b998fb39c4cff712c25 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/843da33b29654b998fb39c4cff712c25 2023-04-27 01:56:42,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9bf8a1bd81044bd095570c13b2325462 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9bf8a1bd81044bd095570c13b2325462 2023-04-27 01:56:42,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3388cfaea35743dd86e1d20718a9f9b2 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/3388cfaea35743dd86e1d20718a9f9b2 2023-04-27 01:56:42,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/7cc6d465fced49e8aff4a655c25ea5cb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/7cc6d465fced49e8aff4a655c25ea5cb 2023-04-27 01:56:42,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/877c33927f4148cb8d94b2ea63d1daee to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/877c33927f4148cb8d94b2ea63d1daee 2023-04-27 01:56:42,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/d75c97259cc64e018f704214056c75b8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/d75c97259cc64e018f704214056c75b8 2023-04-27 01:56:42,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/86b7ad9c6a2744ada258b34ceddebfd6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/86b7ad9c6a2744ada258b34ceddebfd6 2023-04-27 01:56:42,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9f32e4e2bc9f49f0a6e484677c724ae4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/9f32e4e2bc9f49f0a6e484677c724ae4 2023-04-27 01:56:42,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b18c3246154343059c353e3412ade4da to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b18c3246154343059c353e3412ade4da 2023-04-27 01:56:42,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/c8ea7b13dc8241229374968a3e6bd5f4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/c8ea7b13dc8241229374968a3e6bd5f4 2023-04-27 01:56:42,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/78a5b88772d34a5ebecf6c7d88410964 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/78a5b88772d34a5ebecf6c7d88410964 2023-04-27 01:56:42,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11f4003e42a4458a4948c9c63489e9d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/e11f4003e42a4458a4948c9c63489e9d 2023-04-27 01:56:42,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/ea7c54555bd744a8ab311760b687289e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/ea7c54555bd744a8ab311760b687289e 2023-04-27 01:56:42,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/932011fe4ef94ff6aee18e279dea4332 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/932011fe4ef94ff6aee18e279dea4332 2023-04-27 01:56:42,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b17faa3652234988a96d5e8659a89bef to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/b17faa3652234988a96d5e8659a89bef 2023-04-27 01:56:42,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/efc893b9a19345ce835873b22efa3719 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/efc893b9a19345ce835873b22efa3719 2023-04-27 01:56:42,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8c94c2bafba84014b9601ec046b05164 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8c94c2bafba84014b9601ec046b05164 2023-04-27 01:56:42,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/bac67d59b85d486899daeec7dbc3b57a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/bac67d59b85d486899daeec7dbc3b57a 2023-04-27 01:56:42,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/950e050916b145a7b9b48fa002a18629 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/950e050916b145a7b9b48fa002a18629 2023-04-27 01:56:42,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/1ccc4d9acea34cf1adae2c7c7b42930a to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/1ccc4d9acea34cf1adae2c7c7b42930a 2023-04-27 01:56:42,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f66d01f3036a4e16b4f1e6161b0ee543, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/23c42239e84848058ff1df1eb41e4755, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0aecb485b2a04f3f8a71ee5390f018a4, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/a3c674adeb21489b98afe8f5ba00b00f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/56ac11ea5da24837a23ba2b1125b0825, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/b8326f1c1e5d4418af6bc730bb212cab, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1f863289a6bb4f45ab3ce62df57709b5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/ef21d999337641e9be103e80ef625072, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3de56c46de744e1d99bf601d3230e050, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6362e932a3264fb8b27c7e11bdd3a5a7, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bc777593ad314f8885e16e16b0686b08, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/cd9d305721aa472aa21441a68c4f900d, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/85909ff8aaf3424299fea1aa46ffc5e5, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/946d3d6abd074c0c87d16e7dfc971530, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3cabdad74a9740159260149f9a3d77bc, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0de5382f689d4babafc99483a2636360, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/c4b4d72f61a34b909248317b781a4d51, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3e48432f10934a9b9e2c98b4abdefb39, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/be01351e978145a18c38c846abcb9d58, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/321f760edf0349f6b13c65da75c31e59, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/fea8c39711fd4e44b6b8d77e27635bc1, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bd3e097e729d47a5ae4e58077bb75c8e, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/606b2c3ce38c4508824a43237c82d588, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/96f009cca0194d57a3de44697d1a8d89, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f9bd3cc52abb4889ac7a3524a55f30cd, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/5985b139f1944126b112b079e73a764f, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1e21c436851243c39992ae38e2d1c14b, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/db1adb491ce64a4695a3272eb6bc9992, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/4da956f473ee4d189326135aab4d3165, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3c946e12a8874c318fd0d3ee4befcb26, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6df82101450949cfa2faae47a987a091] to archive 2023-04-27 01:56:42,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 01:56:42,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f66d01f3036a4e16b4f1e6161b0ee543 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f66d01f3036a4e16b4f1e6161b0ee543 2023-04-27 01:56:42,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/23c42239e84848058ff1df1eb41e4755 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/23c42239e84848058ff1df1eb41e4755 2023-04-27 01:56:42,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0aecb485b2a04f3f8a71ee5390f018a4 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0aecb485b2a04f3f8a71ee5390f018a4 2023-04-27 01:56:42,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/a3c674adeb21489b98afe8f5ba00b00f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/a3c674adeb21489b98afe8f5ba00b00f 2023-04-27 01:56:42,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/56ac11ea5da24837a23ba2b1125b0825 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/56ac11ea5da24837a23ba2b1125b0825 2023-04-27 01:56:42,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/b8326f1c1e5d4418af6bc730bb212cab to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/b8326f1c1e5d4418af6bc730bb212cab 2023-04-27 01:56:42,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1f863289a6bb4f45ab3ce62df57709b5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1f863289a6bb4f45ab3ce62df57709b5 2023-04-27 01:56:42,627 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/ef21d999337641e9be103e80ef625072 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/ef21d999337641e9be103e80ef625072 2023-04-27 01:56:42,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3de56c46de744e1d99bf601d3230e050 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3de56c46de744e1d99bf601d3230e050 2023-04-27 01:56:42,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6362e932a3264fb8b27c7e11bdd3a5a7 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6362e932a3264fb8b27c7e11bdd3a5a7 2023-04-27 01:56:42,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bc777593ad314f8885e16e16b0686b08 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bc777593ad314f8885e16e16b0686b08 2023-04-27 01:56:42,630 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/cd9d305721aa472aa21441a68c4f900d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/cd9d305721aa472aa21441a68c4f900d 2023-04-27 01:56:42,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/85909ff8aaf3424299fea1aa46ffc5e5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/85909ff8aaf3424299fea1aa46ffc5e5 2023-04-27 01:56:42,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/946d3d6abd074c0c87d16e7dfc971530 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/946d3d6abd074c0c87d16e7dfc971530 2023-04-27 01:56:42,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3cabdad74a9740159260149f9a3d77bc to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3cabdad74a9740159260149f9a3d77bc 2023-04-27 01:56:42,633 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0de5382f689d4babafc99483a2636360 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/0de5382f689d4babafc99483a2636360 2023-04-27 01:56:42,634 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/c4b4d72f61a34b909248317b781a4d51 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/c4b4d72f61a34b909248317b781a4d51 2023-04-27 01:56:42,635 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3e48432f10934a9b9e2c98b4abdefb39 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3e48432f10934a9b9e2c98b4abdefb39 2023-04-27 01:56:42,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/be01351e978145a18c38c846abcb9d58 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/be01351e978145a18c38c846abcb9d58 2023-04-27 01:56:42,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/321f760edf0349f6b13c65da75c31e59 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/321f760edf0349f6b13c65da75c31e59 2023-04-27 01:56:42,637 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/fea8c39711fd4e44b6b8d77e27635bc1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/fea8c39711fd4e44b6b8d77e27635bc1 2023-04-27 01:56:42,638 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bd3e097e729d47a5ae4e58077bb75c8e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/bd3e097e729d47a5ae4e58077bb75c8e 2023-04-27 01:56:42,639 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/606b2c3ce38c4508824a43237c82d588 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/606b2c3ce38c4508824a43237c82d588 2023-04-27 01:56:42,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/96f009cca0194d57a3de44697d1a8d89 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/96f009cca0194d57a3de44697d1a8d89 2023-04-27 01:56:42,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f9bd3cc52abb4889ac7a3524a55f30cd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/f9bd3cc52abb4889ac7a3524a55f30cd 2023-04-27 01:56:42,641 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/5985b139f1944126b112b079e73a764f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/5985b139f1944126b112b079e73a764f 2023-04-27 01:56:42,642 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1e21c436851243c39992ae38e2d1c14b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/1e21c436851243c39992ae38e2d1c14b 2023-04-27 01:56:42,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/db1adb491ce64a4695a3272eb6bc9992 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/db1adb491ce64a4695a3272eb6bc9992 2023-04-27 01:56:42,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/4da956f473ee4d189326135aab4d3165 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/4da956f473ee4d189326135aab4d3165 2023-04-27 01:56:42,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3c946e12a8874c318fd0d3ee4befcb26 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/3c946e12a8874c318fd0d3ee4befcb26 2023-04-27 01:56:42,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6df82101450949cfa2faae47a987a091 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/6df82101450949cfa2faae47a987a091 2023-04-27 01:56:42,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/recovered.edits/467.seqid, newMaxSeqId=467, maxSeqId=4 2023-04-27 01:56:42,650 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627. 2023-04-27 01:56:42,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ec4ba6d393d7d8e622c952e9d8a3e627: 2023-04-27 01:56:42,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:42,652 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=ec4ba6d393d7d8e622c952e9d8a3e627, regionState=CLOSED 2023-04-27 01:56:42,652 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682560602652"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682560602652"}]},"ts":"1682560602652"} 2023-04-27 01:56:42,654 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-27 01:56:42,654 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure ec4ba6d393d7d8e622c952e9d8a3e627, server=jenkins-hbase4.apache.org,38085,1682560417985 in 1.8890 sec 2023-04-27 01:56:42,655 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-27 01:56:42,655 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ec4ba6d393d7d8e622c952e9d8a3e627, UNASSIGN in 1.8920 sec 2023-04-27 01:56:42,655 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682560602655"}]},"ts":"1682560602655"} 2023-04-27 01:56:42,656 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 01:56:42,657 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 01:56:42,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9010 sec 2023-04-27 01:56:42,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 01:56:42,864 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-27 01:56:42,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 01:56:42,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:42,867 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:42,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-27 01:56:42,867 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:42,871 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:42,873 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C, FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/recovered.edits] 2023-04-27 01:56:42,875 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/49852e6d7e8e4f36b3e1b352540314b6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/49852e6d7e8e4f36b3e1b352540314b6 2023-04-27 01:56:42,876 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5fe4fe2b0ec349e391a8ca4aa4b1eda5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/5fe4fe2b0ec349e391a8ca4aa4b1eda5 2023-04-27 01:56:42,877 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b5e95649448e4c0e82373629658c4f56 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/A/b5e95649448e4c0e82373629658c4f56 2023-04-27 01:56:42,879 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/34132b65b9124423a599eafa88b5a43e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/34132b65b9124423a599eafa88b5a43e 2023-04-27 01:56:42,880 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8acc055c99c24953aba99029bbf4ff28 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/8acc055c99c24953aba99029bbf4ff28 2023-04-27 01:56:42,881 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/92192bd39da34539b76187133acd7c42 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/B/92192bd39da34539b76187133acd7c42 2023-04-27 01:56:42,883 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/490592d07b914524999392853a29b340 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/490592d07b914524999392853a29b340 2023-04-27 01:56:42,883 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/674e5e488c57402e9a62c29f0d539f24 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/674e5e488c57402e9a62c29f0d539f24 2023-04-27 01:56:42,884 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/847fa24031fd4a6f860ccb279609e617 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/C/847fa24031fd4a6f860ccb279609e617 2023-04-27 01:56:42,886 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/recovered.edits/467.seqid to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627/recovered.edits/467.seqid 2023-04-27 01:56:42,886 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/.tmp/data/default/TestAcidGuarantees/ec4ba6d393d7d8e622c952e9d8a3e627 2023-04-27 01:56:42,886 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 01:56:42,887 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 01:56:42,887 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-27 01:56:42,889 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042707be735249da4090af0fb81ecb1e04b5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042707be735249da4090af0fb81ecb1e04b5 2023-04-27 01:56:42,890 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270e2d243c22ac4bb99ce5138457b3ed2b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270e2d243c22ac4bb99ce5138457b3ed2b 2023-04-27 01:56:42,891 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427196d5534d9c0470aaf35c522c26f0d20 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427196d5534d9c0470aaf35c522c26f0d20 2023-04-27 01:56:42,891 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427299219d9fdb54797a4eca9d741481e43 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427299219d9fdb54797a4eca9d741481e43 2023-04-27 01:56:42,892 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042737bf67dae7864d5ba7e4b68c5bb71a1e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042737bf67dae7864d5ba7e4b68c5bb71a1e 2023-04-27 01:56:42,893 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042744ffea9f396c4dcf90a0182c367bb3da to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042744ffea9f396c4dcf90a0182c367bb3da 2023-04-27 01:56:42,894 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042750e13cc9255e4a98a40030ae5addf33c to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042750e13cc9255e4a98a40030ae5addf33c 2023-04-27 01:56:42,895 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427542e012d18744d93b26502d745e17a02 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427542e012d18744d93b26502d745e17a02 2023-04-27 01:56:42,895 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275a68cba194bb438aa2ffa79dc19931f8 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275a68cba194bb438aa2ffa79dc19931f8 2023-04-27 01:56:42,896 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276db4a3f4af7e4d73af3919463bdf82e1 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276db4a3f4af7e4d73af3919463bdf82e1 2023-04-27 01:56:42,897 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427724bea67ec484b38af6234c41a6c3c9b to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427724bea67ec484b38af6234c41a6c3c9b 2023-04-27 01:56:42,898 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278403a3eb203a4e6b9839b12341dbf25e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278403a3eb203a4e6b9839b12341dbf25e 2023-04-27 01:56:42,899 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278a45df1817964ba89c90c30da4ac35d6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278a45df1817964ba89c90c30da4ac35d6 2023-04-27 01:56:42,899 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278f2ef3164c4242c7bc68efc3cf5b19a5 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278f2ef3164c4242c7bc68efc3cf5b19a5 2023-04-27 01:56:42,900 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278fc21305be1e483597671ca35ec3528d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278fc21305be1e483597671ca35ec3528d 2023-04-27 01:56:42,901 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b1443f87123d463296578a69c8fe589f to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b1443f87123d463296578a69c8fe589f 2023-04-27 01:56:42,902 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b7b4524fa37f4692b3bef2ed33a59a2e to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b7b4524fa37f4692b3bef2ed33a59a2e 2023-04-27 01:56:42,903 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bb9d1c0040ee4136ba2e6efa5cb119fb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bb9d1c0040ee4136ba2e6efa5cb119fb 2023-04-27 01:56:42,904 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d39bec6f57ca47bb91e1ae1acc7f7f8d to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d39bec6f57ca47bb91e1ae1acc7f7f8d 2023-04-27 01:56:42,905 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d54906720a6a4ba985d1a1fa96e3bbca to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d54906720a6a4ba985d1a1fa96e3bbca 2023-04-27 01:56:42,906 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427da008c8993e3437d9f50a5eded463175 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427da008c8993e3437d9f50a5eded463175 2023-04-27 01:56:42,906 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e10cddcdfeed479fba9346bf38fda714 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e10cddcdfeed479fba9346bf38fda714 2023-04-27 01:56:42,907 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e268024c8d1b419eac7a202131c3cbdb to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e268024c8d1b419eac7a202131c3cbdb 2023-04-27 01:56:42,908 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e2e560428215450c9d8efdbee0ce30a0 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e2e560428215450c9d8efdbee0ce30a0 2023-04-27 01:56:42,909 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f3c03bd14c3d4a2dbcaa44c812a3c103 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f3c03bd14c3d4a2dbcaa44c812a3c103 2023-04-27 01:56:42,910 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f6244a777aa44a81825a9957794477f6 to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f6244a777aa44a81825a9957794477f6 2023-04-27 01:56:42,910 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fd428b9e85664cc6921218715534c8cd to hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fd428b9e85664cc6921218715534c8cd 2023-04-27 01:56:42,911 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 01:56:42,912 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:42,914 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 01:56:42,915 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 01:56:42,916 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:42,916 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 01:56:42,916 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682560602916"}]},"ts":"9223372036854775807"} 2023-04-27 01:56:42,917 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 01:56:42,917 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ec4ba6d393d7d8e622c952e9d8a3e627, NAME => 'TestAcidGuarantees,,1682560572687.ec4ba6d393d7d8e622c952e9d8a3e627.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 01:56:42,917 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 01:56:42,917 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682560602917"}]},"ts":"9223372036854775807"} 2023-04-27 01:56:42,918 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 01:56:42,921 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 01:56:42,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-04-27 01:56:42,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34643] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-27 01:56:42,969 INFO [Listener at localhost/42495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-27 01:56:42,980 INFO [Listener at localhost/42495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=564 (was 558) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 394) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 170), AvailableMemoryMB=12437 (was 12339) - AvailableMemoryMB LEAK? - 2023-04-27 01:56:42,980 INFO [Listener at localhost/42495] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-27 01:56:42,980 INFO [Listener at localhost/42495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 01:56:42,981 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c01f436 to 127.0.0.1:52731 2023-04-27 01:56:42,981 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:42,981 DEBUG [Listener at localhost/42495] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-27 01:56:42,981 DEBUG [Listener at localhost/42495] util.JVMClusterUtil(257): Found active master hash=1120977461, stopped=false 2023-04-27 01:56:42,981 INFO [Listener at localhost/42495] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:56:42,982 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-27 01:56:42,982 INFO [Listener at localhost/42495] procedure2.ProcedureExecutor(629): Stopping 2023-04-27 01:56:42,982 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:56:42,982 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-27 01:56:42,983 DEBUG [Listener at localhost/42495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7731e96a to 127.0.0.1:52731 2023-04-27 01:56:42,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 01:56:42,983 DEBUG [Listener at localhost/42495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:42,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 01:56:42,983 INFO [Listener at localhost/42495] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38085,1682560417985' ***** 2023-04-27 01:56:42,983 INFO [Listener at localhost/42495] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-27 01:56:42,983 INFO [RS:0;jenkins-hbase4:38085] regionserver.HeapMemoryManager(220): Stopping 2023-04-27 01:56:42,984 INFO [RS:0;jenkins-hbase4:38085] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-27 01:56:42,984 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-27 01:56:42,984 INFO [RS:0;jenkins-hbase4:38085] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-27 01:56:42,984 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(3304): Received CLOSE for 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:56:42,985 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:42,985 DEBUG [RS:0;jenkins-hbase4:38085] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x199fd150 to 127.0.0.1:52731 2023-04-27 01:56:42,985 DEBUG [RS:0;jenkins-hbase4:38085] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:42,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8b6eb92cd0f6859befa2a3fe078eb91a, disabling compactions & flushes 2023-04-27 01:56:42,985 INFO [RS:0;jenkins-hbase4:38085] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-27 01:56:42,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:56:42,986 INFO [RS:0;jenkins-hbase4:38085] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-27 01:56:42,986 INFO [RS:0;jenkins-hbase4:38085] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-27 01:56:42,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:56:42,986 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-27 01:56:42,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. after waiting 0 ms 2023-04-27 01:56:42,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:56:42,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8b6eb92cd0f6859befa2a3fe078eb91a 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-27 01:56:42,986 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-27 01:56:42,986 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 8b6eb92cd0f6859befa2a3fe078eb91a=hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a.} 2023-04-27 01:56:42,986 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-27 01:56:42,986 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-27 01:56:42,986 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-27 01:56:42,986 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-27 01:56:42,987 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-27 01:56:42,987 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-27 01:56:42,988 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1505): Waiting on 1588230740, 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:56:43,018 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-27 01:56:43,188 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1505): Waiting on 1588230740, 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:56:43,388 DEBUG [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1505): Waiting on 1588230740, 8b6eb92cd0f6859befa2a3fe078eb91a 2023-04-27 01:56:43,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/.tmp/info/8ddcace865214cd2a4255118c3798ec5 2023-04-27 01:56:43,393 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/.tmp/info/e44b52a84baf467ea67625cc28b5ae85 2023-04-27 01:56:43,398 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e44b52a84baf467ea67625cc28b5ae85 2023-04-27 01:56:43,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/.tmp/info/8ddcace865214cd2a4255118c3798ec5 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/info/8ddcace865214cd2a4255118c3798ec5 2023-04-27 01:56:43,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/info/8ddcace865214cd2a4255118c3798ec5, entries=2, sequenceid=6, filesize=4.8 K 2023-04-27 01:56:43,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 8b6eb92cd0f6859befa2a3fe078eb91a in 419ms, sequenceid=6, compaction requested=false 2023-04-27 01:56:43,407 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/.tmp/rep_barrier/7d77284b538b49f398d8278d031ea6ba 2023-04-27 01:56:43,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/namespace/8b6eb92cd0f6859befa2a3fe078eb91a/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-27 01:56:43,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:56:43,410 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8b6eb92cd0f6859befa2a3fe078eb91a: 2023-04-27 01:56:43,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682560420653.8b6eb92cd0f6859befa2a3fe078eb91a. 2023-04-27 01:56:43,411 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7d77284b538b49f398d8278d031ea6ba 2023-04-27 01:56:43,417 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/.tmp/table/b4c634e5e4fd421c86584f3a974fcf23 2023-04-27 01:56:43,420 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b4c634e5e4fd421c86584f3a974fcf23 2023-04-27 01:56:43,421 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/.tmp/info/e44b52a84baf467ea67625cc28b5ae85 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/info/e44b52a84baf467ea67625cc28b5ae85 2023-04-27 01:56:43,424 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e44b52a84baf467ea67625cc28b5ae85 2023-04-27 01:56:43,424 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/info/e44b52a84baf467ea67625cc28b5ae85, entries=22, sequenceid=93, filesize=7.1 K 2023-04-27 01:56:43,425 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/.tmp/rep_barrier/7d77284b538b49f398d8278d031ea6ba as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/rep_barrier/7d77284b538b49f398d8278d031ea6ba 2023-04-27 01:56:43,428 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7d77284b538b49f398d8278d031ea6ba 2023-04-27 01:56:43,428 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/rep_barrier/7d77284b538b49f398d8278d031ea6ba, entries=6, sequenceid=93, filesize=5.5 K 2023-04-27 01:56:43,428 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/.tmp/table/b4c634e5e4fd421c86584f3a974fcf23 as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/table/b4c634e5e4fd421c86584f3a974fcf23 2023-04-27 01:56:43,431 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b4c634e5e4fd421c86584f3a974fcf23 2023-04-27 01:56:43,431 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/table/b4c634e5e4fd421c86584f3a974fcf23, entries=9, sequenceid=93, filesize=5.5 K 2023-04-27 01:56:43,431 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 444ms, sequenceid=93, compaction requested=false 2023-04-27 01:56:43,436 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-27 01:56:43,436 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-27 01:56:43,438 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-27 01:56:43,438 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-27 01:56:43,438 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-27 01:56:43,589 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38085,1682560417985; all regions closed. 2023-04-27 01:56:43,596 DEBUG [RS:0;jenkins-hbase4:38085] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/oldWALs 2023-04-27 01:56:43,596 INFO [RS:0;jenkins-hbase4:38085] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38085%2C1682560417985.meta:.meta(num 1682560420395) 2023-04-27 01:56:43,601 DEBUG [RS:0;jenkins-hbase4:38085] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/oldWALs 2023-04-27 01:56:43,601 INFO [RS:0;jenkins-hbase4:38085] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38085%2C1682560417985:(num 1682560419863) 2023-04-27 01:56:43,601 DEBUG [RS:0;jenkins-hbase4:38085] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:43,601 INFO [RS:0;jenkins-hbase4:38085] regionserver.LeaseManager(133): Closed leases 2023-04-27 01:56:43,601 INFO [RS:0;jenkins-hbase4:38085] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-27 01:56:43,602 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-27 01:56:43,603 INFO [RS:0;jenkins-hbase4:38085] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38085 2023-04-27 01:56:43,608 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38085,1682560417985 2023-04-27 01:56:43,608 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-27 01:56:43,609 ERROR [Listener at localhost/42495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b8d2e50 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2323] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 01:56:43,609 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-27 01:56:43,609 ERROR [Listener at localhost/42495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@192d1b22 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2323] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 01:56:43,609 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38085,1682560417985] 2023-04-27 01:56:43,610 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38085,1682560417985; numProcessing=1 2023-04-27 01:56:43,613 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38085,1682560417985 already deleted, retry=false 2023-04-27 01:56:43,613 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38085,1682560417985 expired; onlineServers=0 2023-04-27 01:56:43,613 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34643,1682560416787' ***** 2023-04-27 01:56:43,613 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-27 01:56:43,613 DEBUG [M:0;jenkins-hbase4:34643] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3edc80e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-27 01:56:43,613 INFO [M:0;jenkins-hbase4:34643] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34643,1682560416787 2023-04-27 01:56:43,613 INFO [M:0;jenkins-hbase4:34643] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34643,1682560416787; all regions closed. 2023-04-27 01:56:43,613 DEBUG [M:0;jenkins-hbase4:34643] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 01:56:43,613 DEBUG [M:0;jenkins-hbase4:34643] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-27 01:56:43,613 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-27 01:56:43,613 DEBUG [M:0;jenkins-hbase4:34643] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-27 01:56:43,613 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682560419536] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682560419536,5,FailOnTimeoutGroup] 2023-04-27 01:56:43,613 INFO [M:0;jenkins-hbase4:34643] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-27 01:56:43,613 INFO [M:0;jenkins-hbase4:34643] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-27 01:56:43,613 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682560419533] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682560419533,5,FailOnTimeoutGroup] 2023-04-27 01:56:43,614 INFO [M:0;jenkins-hbase4:34643] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-27 01:56:43,614 DEBUG [M:0;jenkins-hbase4:34643] master.HMaster(1512): Stopping service threads 2023-04-27 01:56:43,614 INFO [M:0;jenkins-hbase4:34643] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-27 01:56:43,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-27 01:56:43,614 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 01:56:43,615 ERROR [M:0;jenkins-hbase4:34643] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-27 01:56:43,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-27 01:56:43,615 INFO [M:0;jenkins-hbase4:34643] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-27 01:56:43,615 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-27 01:56:43,616 DEBUG [M:0;jenkins-hbase4:34643] zookeeper.ZKUtil(398): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-27 01:56:43,616 WARN [M:0;jenkins-hbase4:34643] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-27 01:56:43,616 INFO [M:0;jenkins-hbase4:34643] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-27 01:56:43,616 INFO [M:0;jenkins-hbase4:34643] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-27 01:56:43,617 DEBUG [M:0;jenkins-hbase4:34643] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-27 01:56:43,617 INFO [M:0;jenkins-hbase4:34643] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:56:43,617 DEBUG [M:0;jenkins-hbase4:34643] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:56:43,617 DEBUG [M:0;jenkins-hbase4:34643] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-27 01:56:43,617 DEBUG [M:0;jenkins-hbase4:34643] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:56:43,617 INFO [M:0;jenkins-hbase4:34643] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.85 KB heapSize=393.55 KB 2023-04-27 01:56:43,710 INFO [RS:0;jenkins-hbase4:38085] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38085,1682560417985; zookeeper connection closed. 2023-04-27 01:56:43,710 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 01:56:43,711 ERROR [Listener at localhost/42495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@61373815 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2323] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 01:56:43,711 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6b22979e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6b22979e 2023-04-27 01:56:43,711 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): regionserver:38085-0x101c35262760001, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 01:56:43,711 ERROR [Listener at localhost/42495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2b9a2f5e rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2323] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 01:56:43,711 INFO [Listener at localhost/42495] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-27 01:56:44,030 INFO [M:0;jenkins-hbase4:34643] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.85 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f48c99f926c94e1e96a9779d85cc0eaf 2023-04-27 01:56:44,033 INFO [M:0;jenkins-hbase4:34643] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f48c99f926c94e1e96a9779d85cc0eaf 2023-04-27 01:56:44,034 DEBUG [M:0;jenkins-hbase4:34643] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f48c99f926c94e1e96a9779d85cc0eaf as hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f48c99f926c94e1e96a9779d85cc0eaf 2023-04-27 01:56:44,037 INFO [M:0;jenkins-hbase4:34643] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f48c99f926c94e1e96a9779d85cc0eaf 2023-04-27 01:56:44,037 INFO [M:0;jenkins-hbase4:34643] regionserver.HStore(1080): Added hdfs://localhost:44221/user/jenkins/test-data/4fc40a43-ac69-cc6c-f06d-930b30257e0b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f48c99f926c94e1e96a9779d85cc0eaf, entries=65, sequenceid=578, filesize=24.7 K 2023-04-27 01:56:44,038 INFO [M:0;jenkins-hbase4:34643] regionserver.HRegion(2948): Finished flush of dataSize ~344.85 KB/353123, heapSize ~393.54 KB/402984, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=578, compaction requested=false 2023-04-27 01:56:44,038 INFO [M:0;jenkins-hbase4:34643] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 01:56:44,038 DEBUG [M:0;jenkins-hbase4:34643] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-27 01:56:44,041 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-27 01:56:44,041 INFO [M:0;jenkins-hbase4:34643] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-27 01:56:44,042 INFO [M:0;jenkins-hbase4:34643] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34643 2023-04-27 01:56:44,043 DEBUG [M:0;jenkins-hbase4:34643] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34643,1682560416787 already deleted, retry=false 2023-04-27 01:56:44,148 INFO [M:0;jenkins-hbase4:34643] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34643,1682560416787; zookeeper connection closed. 2023-04-27 01:56:44,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 01:56:44,148 ERROR [Listener at localhost/42495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@448ec791 rejected from java.util.concurrent.ThreadPoolExecutor@6b73b90d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3884] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 01:56:44,148 DEBUG [Listener at localhost/42495-EventThread] zookeeper.ZKWatcher(600): master:34643-0x101c35262760000, quorum=127.0.0.1:52731, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 01:56:44,148 ERROR [Listener at localhost/42495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@343a90d0 rejected from java.util.concurrent.ThreadPoolExecutor@6b73b90d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3884] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 01:56:44,150 WARN [Listener at localhost/42495] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-27 01:56:44,154 INFO [Listener at localhost/42495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-27 01:56:44,263 WARN [BP-809979935-172.31.14.131-1682560413421 heartbeating to localhost/127.0.0.1:44221] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-27 01:56:44,263 WARN [BP-809979935-172.31.14.131-1682560413421 heartbeating to localhost/127.0.0.1:44221] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-809979935-172.31.14.131-1682560413421 (Datanode Uuid 82e9d633-1016-4634-b79e-7e0d7bfee16d) service to localhost/127.0.0.1:44221 2023-04-27 01:56:44,267 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/cluster_f8bfb6d3-6781-ae77-a03b-b65e0afd0224/dfs/data/data1/current/BP-809979935-172.31.14.131-1682560413421] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-27 01:56:44,267 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1b1ad6e1-6c1a-484d-26be-f4360eee76d7/cluster_f8bfb6d3-6781-ae77-a03b-b65e0afd0224/dfs/data/data2/current/BP-809979935-172.31.14.131-1682560413421] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-27 01:56:44,306 INFO [Listener at localhost/42495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-27 01:56:44,470 INFO [Listener at localhost/42495] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-27 01:56:44,631 INFO [Listener at localhost/42495] hbase.HBaseTestingUtility(1293): Minicluster is down